[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:47:08 dist-test-slave-lzx3 krb5kdc[778](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:47:08 dist-test-slave-lzx3 krb5kdc[778](info): set up 2 sockets
Jun 21 07:47:08 dist-test-slave-lzx3 krb5kdc[778](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260621 07:47:09.488852 771 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/krb5kdc/test-user.keytab.
Jun 21 07:47:09 dist-test-slave-lzx3 krb5kdc[778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:47:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:47:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.192.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
I20260621 07:47:09.594936 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:36631
--webserver_interface=127.0.192.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:42547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:36631
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:09.693754 791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:09.693992 791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:09.694053 791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:09.697537 791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:47:09.697605 791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:09.697639 791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:09.697690 791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:47:09.697723 791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:47:09.702479 791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:42547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:36631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:36631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:09.703306 791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:09.704195 791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:09.712021 797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:09.712029 799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:09.712029 796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:09.712167 791 server_base.cc:1034] running on GCE node
I20260621 07:47:09.712844 791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:09.713737 791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:09.714905 791 hybrid_clock.cc:648] HybridClock initialized: now 1782028029714869 us; error 47 us; skew 500 ppm
Jun 21 07:47:09 dist-test-slave-lzx3 krb5kdc[778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:09.717271 791 init.cc:377] Logged in from keytab as kudu/127.0.192.254@KRBTEST.COM (short username kudu)
I20260621 07:47:09.717970 791 webserver.cc:460] Webserver started at http://127.0.192.254:37507/ using document root <none> and password file <none>
I20260621 07:47:09.718380 791 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:09.718428 791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:09.718598 791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:09.720356 791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "3a2d3a33b8e94722979dc7c6b8338dda"
format_stamp: "Formatted at 2026-06-21 07:47:09 on dist-test-slave-lzx3"
server_key: "a3f328b137b21cd2e6fa267d8d04ab54"
server_key_iv: "e322b5a134de2cbd74c605e7f7c88a25"
server_key_version: "encryptionkey@0"
I20260621 07:47:09.720934 791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "3a2d3a33b8e94722979dc7c6b8338dda"
format_stamp: "Formatted at 2026-06-21 07:47:09 on dist-test-slave-lzx3"
server_key: "a3f328b137b21cd2e6fa267d8d04ab54"
server_key_iv: "e322b5a134de2cbd74c605e7f7c88a25"
server_key_version: "encryptionkey@0"
I20260621 07:47:09.724051 791 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:47:09.726191 806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:09.727214 791 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260621 07:47:09.727329 791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "3a2d3a33b8e94722979dc7c6b8338dda"
format_stamp: "Formatted at 2026-06-21 07:47:09 on dist-test-slave-lzx3"
server_key: "a3f328b137b21cd2e6fa267d8d04ab54"
server_key_iv: "e322b5a134de2cbd74c605e7f7c88a25"
server_key_version: "encryptionkey@0"
I20260621 07:47:09.727417 791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:09.745765 791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:09.747931 791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:09.748107 791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:09.754315 791 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:36631
I20260621 07:47:09.754338 858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:36631 every 8 connection(s)
I20260621 07:47:09.755043 791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:47:09.757270 859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:09.760205 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 791
I20260621 07:47:09.760324 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:47:09.761608 771 external_mini_cluster.cc:1428] Setting key 89d9029b1d9836f8ccd00c57a72e817e
I20260621 07:47:09.762290 859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a2d3a33b8e94722979dc7c6b8338dda: Bootstrap starting.
I20260621 07:47:09.764914 859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a2d3a33b8e94722979dc7c6b8338dda: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:09.765791 859 log.cc:824] T 00000000000000000000000000000000 P 3a2d3a33b8e94722979dc7c6b8338dda: Log is configured to *not* fsync() on all Append() calls
I20260621 07:47:09.767422 859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a2d3a33b8e94722979dc7c6b8338dda: No bootstrap required, opened a new log
I20260621 07:47:09.770924 859 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a2d3a33b8e94722979dc7c6b8338dda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a2d3a33b8e94722979dc7c6b8338dda" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 36631 } }
I20260621 07:47:09.771116 859 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a2d3a33b8e94722979dc7c6b8338dda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:09.771175 859 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a2d3a33b8e94722979dc7c6b8338dda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a2d3a33b8e94722979dc7c6b8338dda, State: Initialized, Role: FOLLOWER
I20260621 07:47:09.771585 859 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a2d3a33b8e94722979dc7c6b8338dda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a2d3a33b8e94722979dc7c6b8338dda" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 36631 } }
I20260621 07:47:09.771696 859 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a2d3a33b8e94722979dc7c6b8338dda [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:47:09.771780 859 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a2d3a33b8e94722979dc7c6b8338dda [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:47:09.771899 859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a2d3a33b8e94722979dc7c6b8338dda [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:09.772734 859 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a2d3a33b8e94722979dc7c6b8338dda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a2d3a33b8e94722979dc7c6b8338dda" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 36631 } }
I20260621 07:47:09.773030 859 leader_election.cc:302] T 00000000000000000000000000000000 P 3a2d3a33b8e94722979dc7c6b8338dda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a2d3a33b8e94722979dc7c6b8338dda; no voters:
I20260621 07:47:09.773281 859 leader_election.cc:288] T 00000000000000000000000000000000 P 3a2d3a33b8e94722979dc7c6b8338dda [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:47:09.773373 864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a2d3a33b8e94722979dc7c6b8338dda [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:47:09.773584 864 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a2d3a33b8e94722979dc7c6b8338dda [term 1 LEADER]: Becoming Leader. State: Replica: 3a2d3a33b8e94722979dc7c6b8338dda, State: Running, Role: LEADER
I20260621 07:47:09.773861 864 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a2d3a33b8e94722979dc7c6b8338dda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a2d3a33b8e94722979dc7c6b8338dda" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 36631 } }
I20260621 07:47:09.774355 859 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a2d3a33b8e94722979dc7c6b8338dda [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:47:09.775274 865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a2d3a33b8e94722979dc7c6b8338dda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a2d3a33b8e94722979dc7c6b8338dda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a2d3a33b8e94722979dc7c6b8338dda" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 36631 } } }
I20260621 07:47:09.775303 866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a2d3a33b8e94722979dc7c6b8338dda [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a2d3a33b8e94722979dc7c6b8338dda. Latest consensus state: current_term: 1 leader_uuid: "3a2d3a33b8e94722979dc7c6b8338dda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a2d3a33b8e94722979dc7c6b8338dda" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 36631 } } }
I20260621 07:47:09.775377 865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a2d3a33b8e94722979dc7c6b8338dda [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:09.775390 866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a2d3a33b8e94722979dc7c6b8338dda [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:09.775743 873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:47:09.778729 873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 21 07:47:09 dist-test-slave-lzx3 krb5kdc[778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:47:09.785521 861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:09.764324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34562 (local address 127.0.192.254:36631)
0621 07:47:09.764723 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:09.764742 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:09.765918 (+ 1176us) server_negotiation.cc:407] Connection header received
0621 07:47:09.769352 (+ 3434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:09.769367 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:09.769625 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:09.769935 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:09.772185 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:09.773234 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:09.774687 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:09.775365 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:09.775515 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:09.780218 (+ 4703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:09.780224 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:09.780240 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:09.780276 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:09.782382 (+ 2106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:09.783029 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:09.783034 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:09.783039 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:09.783171 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:09.783555 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:09.783558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:09.783559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:09.783914 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:09.784136 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:09.784624 (+ 488us) server_negotiation.cc:299] Negotiation successful
0621 07:47:09.784836 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":129,"threads_started":1}
I20260621 07:47:09.786103 873 catalog_manager.cc:1347] Generated new cluster ID: 94b6be2ab45c4dbca3f8e4e381239169
I20260621 07:47:09.786154 873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:47:09.806342 873 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:47:09.807307 873 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:47:09.819024 873 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a2d3a33b8e94722979dc7c6b8338dda: Generated new TSK 0
I20260621 07:47:09.819550 873 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.192.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab.
Entry for principal kudu/127.0.192.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab.
I20260621 07:47:09.849397 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.193:0
--local_ip_for_outbound_sockets=127.0.192.193
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:36631
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:42547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:09.947710 885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:09.947914 885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:09.947975 885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:09.951349 885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:09.951395 885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:09.951505 885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.193
I20260621 07:47:09.956631 885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:42547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:36631
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.193
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:09.957430 885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:09.958238 885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:09.966938 890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:09.966992 891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:09.966938 893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:09.967058 885 server_base.cc:1034] running on GCE node
I20260621 07:47:09.967661 885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:09.968211 885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:09.969379 885 hybrid_clock.cc:648] HybridClock initialized: now 1782028029969350 us; error 41 us; skew 500 ppm
Jun 21 07:47:09 dist-test-slave-lzx3 krb5kdc[778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:09.971810 885 init.cc:377] Logged in from keytab as kudu/127.0.192.193@KRBTEST.COM (short username kudu)
I20260621 07:47:09.972669 885 webserver.cc:460] Webserver started at http://127.0.192.193:37225/ using document root <none> and password file <none>
I20260621 07:47:09.973165 885 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:09.973249 885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:09.973455 885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:09.975366 885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-0/data/instance:
uuid: "cded4470fefb45a2a8314a50c900e08f"
format_stamp: "Formatted at 2026-06-21 07:47:09 on dist-test-slave-lzx3"
server_key: "f0b3d3b7a8b9908be89a0f1c01aba49c"
server_key_iv: "a36c14cc19a8b8304f40bd34ff4e19f5"
server_key_version: "encryptionkey@0"
I20260621 07:47:09.975888 885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-0/wal/instance:
uuid: "cded4470fefb45a2a8314a50c900e08f"
format_stamp: "Formatted at 2026-06-21 07:47:09 on dist-test-slave-lzx3"
server_key: "f0b3d3b7a8b9908be89a0f1c01aba49c"
server_key_iv: "a36c14cc19a8b8304f40bd34ff4e19f5"
server_key_version: "encryptionkey@0"
I20260621 07:47:09.979339 885 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260621 07:47:09.981680 900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:09.982746 885 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 07:47:09.982887 885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-0/wal
uuid: "cded4470fefb45a2a8314a50c900e08f"
format_stamp: "Formatted at 2026-06-21 07:47:09 on dist-test-slave-lzx3"
server_key: "f0b3d3b7a8b9908be89a0f1c01aba49c"
server_key_iv: "a36c14cc19a8b8304f40bd34ff4e19f5"
server_key_version: "encryptionkey@0"
I20260621 07:47:09.982990 885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:09.995023 885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:09.997517 885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:09.997705 885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:09.998215 885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:47:09.998929 885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:47:09.998993 885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:09.999043 885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:47:09.999109 885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:10.007155 885 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.193:42225
I20260621 07:47:10.007177 1013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.193:42225 every 8 connection(s)
I20260621 07:47:10.007833 885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
I20260621 07:47:10.014267 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 885
I20260621 07:47:10.014374 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-0/wal/instance
I20260621 07:47:10.014634 771 external_mini_cluster.cc:1428] Setting key da99f99d8293baa1c2b025362b818eb6
Jun 21 07:47:10 dist-test-slave-lzx3 krb5kdc[778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:47:10.019941 861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:10.009433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:39995 (local address 127.0.192.254:36631)
0621 07:47:10.009589 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:10.009594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:10.010287 (+ 693us) server_negotiation.cc:407] Connection header received
0621 07:47:10.011403 (+ 1116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:10.011405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:10.011449 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:10.011521 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:10.012521 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:10.013107 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:10.014079 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:10.014277 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:10.014380 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:10.016909 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:10.016914 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:10.016920 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:10.016940 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:10.018189 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:10.018614 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:10.018619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:10.018620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:10.018695 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:10.019001 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:10.019005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:10.019007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:10.019164 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:10.019268 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:10.019670 (+ 402us) server_negotiation.cc:299] Negotiation successful
0621 07:47:10.019774 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 07:47:10.020642 1016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:10.009765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:36631 (local address 127.0.192.193:39995)
0621 07:47:10.010157 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:10.010192 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:10.011197 (+ 1005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:10.011666 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:10.011672 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:10.011976 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:10.012392 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:10.012404 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:10.013208 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:10.013211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:10.013977 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:10.013985 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:10.014412 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:10.014418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:10.014596 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:10.015155 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:10.015169 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:10.016771 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:10.018291 (+ 1520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:10.018296 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:10.018307 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:10.018518 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:10.018777 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:10.018780 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:10.018781 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:10.018913 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:10.019267 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:10.019272 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:10.019466 (+ 194us) client_negotiation.cc:769] Sending connection context
0621 07:47:10.019632 (+ 166us) client_negotiation.cc:240] Negotiation successful
0621 07:47:10.019895 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":120,"threads_started":1}
I20260621 07:47:10.021668 1014 heartbeater.cc:347] Connected to a master server at 127.0.192.254:36631
I20260621 07:47:10.021869 1014 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:10.022274 1014 heartbeater.cc:511] Master 127.0.192.254:36631 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.192.194@KRBTEST.COM; defaulting to no policy
I20260621 07:47:10.023594 823 ts_manager.cc:194] Registered new tserver with Master: cded4470fefb45a2a8314a50c900e08f (127.0.192.193:42225)
I20260621 07:47:10.024714 823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.193@KRBTEST.COM'} at 127.0.192.193:39995
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab.
Entry for principal kudu/127.0.192.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab.
I20260621 07:47:10.039845 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.194:0
--local_ip_for_outbound_sockets=127.0.192.194
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:36631
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:42547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:10.137504 1019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:10.137748 1019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:10.137799 1019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:10.141474 1019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:10.141520 1019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:10.141605 1019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.194
I20260621 07:47:10.146113 1019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:42547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:36631
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.194
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:10.146893 1019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:10.147728 1019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 07:47:10.156715 1019 server_base.cc:1034] running on GCE node
W20260621 07:47:10.156749 1024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:10.156749 1025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:10.156776 1027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:10.157233 1019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:10.157768 1019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:10.158948 1019 hybrid_clock.cc:648] HybridClock initialized: now 1782028030158933 us; error 30 us; skew 500 ppm
Jun 21 07:47:10 dist-test-slave-lzx3 krb5kdc[778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:10.161557 1019 init.cc:377] Logged in from keytab as kudu/127.0.192.194@KRBTEST.COM (short username kudu)
I20260621 07:47:10.162369 1019 webserver.cc:460] Webserver started at http://127.0.192.194:37737/ using document root <none> and password file <none>
I20260621 07:47:10.162885 1019 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:10.162976 1019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:10.163208 1019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:10.165123 1019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-1/data/instance:
uuid: "58b65167f9cf461e8c94b3cc92e2d6a4"
format_stamp: "Formatted at 2026-06-21 07:47:10 on dist-test-slave-lzx3"
server_key: "19b61dfe9ba1b43ee6d1b34feafa8866"
server_key_iv: "92770f2c531e35b2f4391cc43d4d0261"
server_key_version: "encryptionkey@0"
I20260621 07:47:10.165649 1019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-1/wal/instance:
uuid: "58b65167f9cf461e8c94b3cc92e2d6a4"
format_stamp: "Formatted at 2026-06-21 07:47:10 on dist-test-slave-lzx3"
server_key: "19b61dfe9ba1b43ee6d1b34feafa8866"
server_key_iv: "92770f2c531e35b2f4391cc43d4d0261"
server_key_version: "encryptionkey@0"
I20260621 07:47:10.168704 1019 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260621 07:47:10.170634 1034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:10.171716 1019 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:10.171840 1019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-1/wal
uuid: "58b65167f9cf461e8c94b3cc92e2d6a4"
format_stamp: "Formatted at 2026-06-21 07:47:10 on dist-test-slave-lzx3"
server_key: "19b61dfe9ba1b43ee6d1b34feafa8866"
server_key_iv: "92770f2c531e35b2f4391cc43d4d0261"
server_key_version: "encryptionkey@0"
I20260621 07:47:10.171937 1019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:10.189333 1019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:10.191651 1019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:10.191824 1019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:10.192317 1019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:47:10.193063 1019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:47:10.193130 1019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:10.193183 1019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:47:10.193218 1019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:10.201848 1019 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.194:39231
I20260621 07:47:10.201876 1147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.194:39231 every 8 connection(s)
I20260621 07:47:10.202613 1019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
I20260621 07:47:10.204345 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 1019
I20260621 07:47:10.204487 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-1/wal/instance
I20260621 07:47:10.204871 771 external_mini_cluster.cc:1428] Setting key 339c37d4b18b9e14ccfb9965c0d0a24c
Jun 21 07:47:10 dist-test-slave-lzx3 krb5kdc[778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.192.195@KRBTEST.COM; defaulting to no policy
I20260621 07:47:10.215891 861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:10.204024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:46713 (local address 127.0.192.254:36631)
0621 07:47:10.204165 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:10.204170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:10.205013 (+ 843us) server_negotiation.cc:407] Connection header received
0621 07:47:10.206538 (+ 1525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:10.206540 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:10.206579 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:10.206637 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:10.207985 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:10.208806 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:10.209773 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:10.210123 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:10.210227 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:10.212750 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:10.212755 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:10.212758 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:10.212785 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:10.214264 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:10.214777 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:10.214782 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:10.214785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:10.214857 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:10.215108 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:10.215110 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:10.215111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:10.215244 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:10.215308 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:10.215695 (+ 387us) server_negotiation.cc:299] Negotiation successful
0621 07:47:10.215796 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 07:47:10.216661 1150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:10.204338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:36631 (local address 127.0.192.194:46713)
0621 07:47:10.204870 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:10.204923 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:10.206348 (+ 1425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:10.206781 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:10.206791 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:10.207212 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:10.207820 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:10.207843 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:10.208904 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:10.208906 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:10.209657 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:10.209665 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:10.210246 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:10.210249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:10.210420 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:10.210907 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:10.210920 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:10.212594 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:10.214372 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:10.214376 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:10.214396 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:10.214690 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:10.214930 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:10.214932 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:10.214933 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:10.215037 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:10.215316 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:10.215320 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:10.215515 (+ 195us) client_negotiation.cc:769] Sending connection context
0621 07:47:10.215640 (+ 125us) client_negotiation.cc:240] Negotiation successful
0621 07:47:10.215859 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":178,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.195@KRBTEST.COM" created.
I20260621 07:47:10.217702 1148 heartbeater.cc:347] Connected to a master server at 127.0.192.254:36631
I20260621 07:47:10.217893 1148 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:10.218406 1148 heartbeater.cc:511] Master 127.0.192.254:36631 requested a full tablet report, sending...
I20260621 07:47:10.219292 823 ts_manager.cc:194] Registered new tserver with Master: 58b65167f9cf461e8c94b3cc92e2d6a4 (127.0.192.194:39231)
I20260621 07:47:10.219858 823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.194@KRBTEST.COM'} at 127.0.192.194:46713
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab.
Entry for principal kudu/127.0.192.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab.
I20260621 07:47:10.231434 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.195:0
--local_ip_for_outbound_sockets=127.0.192.195
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:36631
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:42547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:10.327275 1153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:10.327481 1153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:10.327538 1153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:10.331014 1153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:10.331073 1153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:10.331161 1153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.195
I20260621 07:47:10.335812 1153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:42547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:36631
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.195
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:10.336599 1153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:10.337393 1153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:10.345067 1159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:10.345067 1161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:10.345082 1158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:10.345366 1153 server_base.cc:1034] running on GCE node
I20260621 07:47:10.345731 1153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:10.346191 1153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:10.347357 1153 hybrid_clock.cc:648] HybridClock initialized: now 1782028030347337 us; error 36 us; skew 500 ppm
Jun 21 07:47:10 dist-test-slave-lzx3 krb5kdc[778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:10.349658 1153 init.cc:377] Logged in from keytab as kudu/127.0.192.195@KRBTEST.COM (short username kudu)
I20260621 07:47:10.350325 1153 webserver.cc:460] Webserver started at http://127.0.192.195:36115/ using document root <none> and password file <none>
I20260621 07:47:10.350786 1153 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:10.350867 1153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:10.351059 1153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:10.352840 1153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-2/data/instance:
uuid: "fe2cab649a404c87b3bd7ad82f91f890"
format_stamp: "Formatted at 2026-06-21 07:47:10 on dist-test-slave-lzx3"
server_key: "8c838a4c7c708d2a187aa084a3e60f70"
server_key_iv: "909af8d481d1748a033707a06fcf279e"
server_key_version: "encryptionkey@0"
I20260621 07:47:10.353302 1153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-2/wal/instance:
uuid: "fe2cab649a404c87b3bd7ad82f91f890"
format_stamp: "Formatted at 2026-06-21 07:47:10 on dist-test-slave-lzx3"
server_key: "8c838a4c7c708d2a187aa084a3e60f70"
server_key_iv: "909af8d481d1748a033707a06fcf279e"
server_key_version: "encryptionkey@0"
I20260621 07:47:10.356498 1153 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:47:10.358345 1168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:10.359395 1153 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:10.359529 1153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-2/wal
uuid: "fe2cab649a404c87b3bd7ad82f91f890"
format_stamp: "Formatted at 2026-06-21 07:47:10 on dist-test-slave-lzx3"
server_key: "8c838a4c7c708d2a187aa084a3e60f70"
server_key_iv: "909af8d481d1748a033707a06fcf279e"
server_key_version: "encryptionkey@0"
I20260621 07:47:10.359638 1153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:10.383458 1153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:10.385888 1153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:10.386051 1153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:10.386574 1153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:47:10.387286 1153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:47:10.387348 1153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:10.387398 1153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:47:10.387439 1153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:10.395301 1153 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.195:38501
I20260621 07:47:10.395371 1281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.195:38501 every 8 connection(s)
I20260621 07:47:10.396037 1153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
I20260621 07:47:10.396250 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 1153
I20260621 07:47:10.396355 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782028028522580-771-0/minicluster-data/ts-2/wal/instance
I20260621 07:47:10.396700 771 external_mini_cluster.cc:1428] Setting key a6a9a066565aa70032508aae89cc255a
Jun 21 07:47:10 dist-test-slave-lzx3 krb5kdc[778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:47:10.408319 861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:10.397702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:52653 (local address 127.0.192.254:36631)
0621 07:47:10.397813 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:10.397817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:10.398572 (+ 755us) server_negotiation.cc:407] Connection header received
0621 07:47:10.400183 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:10.400186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:10.400236 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:10.400312 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:10.401406 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:10.401924 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:10.403006 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:10.403217 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:10.403326 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:10.405429 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:10.405432 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:10.405434 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:10.405462 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:10.406564 (+ 1102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:10.407019 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:10.407023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:10.407025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:10.407099 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:10.407421 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:10.407425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:10.407427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:10.407582 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:10.407689 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:10.408077 (+ 388us) server_negotiation.cc:299] Negotiation successful
0621 07:47:10.408182 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260621 07:47:10.409018 1284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:10.398035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:36631 (local address 127.0.192.195:52653)
0621 07:47:10.398442 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:10.398483 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:10.399993 (+ 1510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:10.400469 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:10.400475 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:10.400855 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:10.401252 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:10.401263 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:10.402046 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:10.402049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:10.402888 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:10.402896 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:10.403354 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:10.403357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:10.403511 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:10.403949 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:10.403963 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:10.405256 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:10.406678 (+ 1422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:10.406683 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:10.406695 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:10.406916 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:10.407202 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:10.407205 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:10.407206 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:10.407324 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:10.407682 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:10.407687 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:10.407879 (+ 192us) client_negotiation.cc:769] Sending connection context
0621 07:47:10.408031 (+ 152us) client_negotiation.cc:240] Negotiation successful
0621 07:47:10.408274 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":95,"threads_started":1}
I20260621 07:47:10.409924 1282 heartbeater.cc:347] Connected to a master server at 127.0.192.254:36631
I20260621 07:47:10.410086 1282 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:10.410485 1282 heartbeater.cc:511] Master 127.0.192.254:36631 requested a full tablet report, sending...
I20260621 07:47:10.411468 823 ts_manager.cc:194] Registered new tserver with Master: fe2cab649a404c87b3bd7ad82f91f890 (127.0.192.195:38501)
I20260621 07:47:10.411911 771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:47:10.412202 823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.195@KRBTEST.COM'} at 127.0.192.195:52653
Jun 21 07:47:10 dist-test-slave-lzx3 krb5kdc[778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028030, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 07:47:10 dist-test-slave-lzx3 krb5kdc[778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028030, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.192.193@KRBTEST.COM
I20260621 07:47:10.434685 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:10.426522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48086 (local address 127.0.192.193:42225)
0621 07:47:10.426780 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:10.426790 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:10.426812 (+ 22us) server_negotiation.cc:407] Connection header received
0621 07:47:10.426859 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:10.426863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:10.427000 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:10.427135 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:10.427705 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:10.428338 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:10.429143 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:10.429454 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:10.429565 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:10.431826 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:10.431834 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:10.431843 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:10.431864 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:10.433172 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:10.433558 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:10.433567 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:10.433568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:10.433635 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:10.433872 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:10.433876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:10.433878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:10.434065 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:10.434267 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:10.434417 (+ 150us) server_negotiation.cc:299] Negotiation successful
0621 07:47:10.434539 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":71,"threads_started":1}
W20260621 07:47:10.435410 928 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48086
Jun 21 07:47:10 dist-test-slave-lzx3 krb5kdc[778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 07:47:10 dist-test-slave-lzx3 krb5kdc[778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.192.193@KRBTEST.COM
I20260621 07:47:10.456761 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:10.449296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48096 (local address 127.0.192.193:42225)
0621 07:47:10.449437 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:10.449441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:10.449558 (+ 117us) server_negotiation.cc:407] Connection header received
0621 07:47:10.449701 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:10.449703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:10.449743 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:10.449825 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:10.450392 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:10.450834 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:10.451601 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:10.451824 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:10.451934 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:10.454034 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:10.454039 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:10.454046 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:10.454065 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:10.455178 (+ 1113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:10.455537 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:10.455540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:10.455541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:10.455631 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:10.455940 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:10.455943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:10.455945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:10.456110 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:10.456249 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:10.456480 (+ 231us) server_negotiation.cc:299] Negotiation successful
0621 07:47:10.456613 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 07:47:10.457635 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 885
I20260621 07:47:10.463065 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 1019
I20260621 07:47:10.470052 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 1153
I20260621 07:47:10.474736 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 791
2026-06-21T07:47:10Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1948 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:47:10 dist-test-slave-lzx3 krb5kdc[1296](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:47:10 dist-test-slave-lzx3 krb5kdc[1296](info): set up 2 sockets
Jun 21 07:47:10 dist-test-slave-lzx3 krb5kdc[1296](info): commencing operation
krb5kdc: starting...
W20260621 07:47:11.419914 771 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/krb5kdc/test-user.keytab.
Jun 21 07:47:11 dist-test-slave-lzx3 krb5kdc[1296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:47:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:47:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.192.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
I20260621 07:47:11.527114 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:45639
--webserver_interface=127.0.192.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:43159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:45639
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:11.625308 1309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:11.625495 1309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:11.625526 1309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:11.628715 1309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:47:11.628748 1309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:11.628765 1309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:11.628778 1309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:47:11.628791 1309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:47:11.633313 1309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:43159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:45639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:45639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:11.634020 1309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:11.634728 1309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:11.641682 1314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:11.641670 1315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:11.641806 1309 server_base.cc:1034] running on GCE node
W20260621 07:47:11.641671 1317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:11.642484 1309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:11.643391 1309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:11.644536 1309 hybrid_clock.cc:648] HybridClock initialized: now 1782028031644502 us; error 45 us; skew 500 ppm
Jun 21 07:47:11 dist-test-slave-lzx3 krb5kdc[1296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:11.646976 1309 init.cc:377] Logged in from keytab as kudu/127.0.192.254@KRBTEST.COM (short username kudu)
I20260621 07:47:11.647718 1309 webserver.cc:460] Webserver started at http://127.0.192.254:40935/ using document root <none> and password file <none>
I20260621 07:47:11.648166 1309 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:11.648244 1309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:11.648440 1309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:11.650101 1309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "e19e8bb02b694d3c9b0ff9209bff085b"
format_stamp: "Formatted at 2026-06-21 07:47:11 on dist-test-slave-lzx3"
server_key: "61ad1c5e9c162414fda6746c56709262"
server_key_iv: "17345dde385860f65ae0e172de0796dc"
server_key_version: "encryptionkey@0"
I20260621 07:47:11.650599 1309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "e19e8bb02b694d3c9b0ff9209bff085b"
format_stamp: "Formatted at 2026-06-21 07:47:11 on dist-test-slave-lzx3"
server_key: "61ad1c5e9c162414fda6746c56709262"
server_key_iv: "17345dde385860f65ae0e172de0796dc"
server_key_version: "encryptionkey@0"
I20260621 07:47:11.653800 1309 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:47:11.655779 1324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:11.656787 1309 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:11.656905 1309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "e19e8bb02b694d3c9b0ff9209bff085b"
format_stamp: "Formatted at 2026-06-21 07:47:11 on dist-test-slave-lzx3"
server_key: "61ad1c5e9c162414fda6746c56709262"
server_key_iv: "17345dde385860f65ae0e172de0796dc"
server_key_version: "encryptionkey@0"
I20260621 07:47:11.656997 1309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:11.686524 1309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:11.688962 1309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:11.689131 1309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:11.695454 1309 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:45639
I20260621 07:47:11.695469 1376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:45639 every 8 connection(s)
I20260621 07:47:11.696345 1309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:47:11.698757 1377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:11.701814 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 1309
I20260621 07:47:11.701947 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:47:11.702183 771 external_mini_cluster.cc:1428] Setting key 4b873674b63c0e3ed78c5e467c5ab848
I20260621 07:47:11.704126 1377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e19e8bb02b694d3c9b0ff9209bff085b: Bootstrap starting.
I20260621 07:47:11.706363 1377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e19e8bb02b694d3c9b0ff9209bff085b: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:11.707086 1377 log.cc:824] T 00000000000000000000000000000000 P e19e8bb02b694d3c9b0ff9209bff085b: Log is configured to *not* fsync() on all Append() calls
I20260621 07:47:11.708921 1377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e19e8bb02b694d3c9b0ff9209bff085b: No bootstrap required, opened a new log
Jun 21 07:47:11 dist-test-slave-lzx3 krb5kdc[1296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:47:11.712005 1377 raft_consensus.cc:348] T 00000000000000000000000000000000 P e19e8bb02b694d3c9b0ff9209bff085b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19e8bb02b694d3c9b0ff9209bff085b" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45639 } }
I20260621 07:47:11.712188 1377 raft_consensus.cc:374] T 00000000000000000000000000000000 P e19e8bb02b694d3c9b0ff9209bff085b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:11.712219 1377 raft_consensus.cc:729] T 00000000000000000000000000000000 P e19e8bb02b694d3c9b0ff9209bff085b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e19e8bb02b694d3c9b0ff9209bff085b, State: Initialized, Role: FOLLOWER
I20260621 07:47:11.712699 1377 consensus_queue.cc:260] T 00000000000000000000000000000000 P e19e8bb02b694d3c9b0ff9209bff085b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19e8bb02b694d3c9b0ff9209bff085b" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45639 } }
I20260621 07:47:11.712858 1377 raft_consensus.cc:388] T 00000000000000000000000000000000 P e19e8bb02b694d3c9b0ff9209bff085b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:47:11.712926 1377 raft_consensus.cc:482] T 00000000000000000000000000000000 P e19e8bb02b694d3c9b0ff9209bff085b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:47:11.713052 1377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e19e8bb02b694d3c9b0ff9209bff085b [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:11.713775 1377 raft_consensus.cc:504] T 00000000000000000000000000000000 P e19e8bb02b694d3c9b0ff9209bff085b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19e8bb02b694d3c9b0ff9209bff085b" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45639 } }
I20260621 07:47:11.714085 1377 leader_election.cc:302] T 00000000000000000000000000000000 P e19e8bb02b694d3c9b0ff9209bff085b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e19e8bb02b694d3c9b0ff9209bff085b; no voters:
I20260621 07:47:11.714336 1377 leader_election.cc:288] T 00000000000000000000000000000000 P e19e8bb02b694d3c9b0ff9209bff085b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:47:11.714389 1382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e19e8bb02b694d3c9b0ff9209bff085b [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:47:11.714560 1382 raft_consensus.cc:686] T 00000000000000000000000000000000 P e19e8bb02b694d3c9b0ff9209bff085b [term 1 LEADER]: Becoming Leader. State: Replica: e19e8bb02b694d3c9b0ff9209bff085b, State: Running, Role: LEADER
I20260621 07:47:11.714843 1382 consensus_queue.cc:237] T 00000000000000000000000000000000 P e19e8bb02b694d3c9b0ff9209bff085b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19e8bb02b694d3c9b0ff9209bff085b" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45639 } }
I20260621 07:47:11.715333 1380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:11.703421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59164 (local address 127.0.192.254:45639)
0621 07:47:11.703799 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:11.703814 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:11.703846 (+ 32us) server_negotiation.cc:407] Connection header received
0621 07:47:11.704453 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:11.704469 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:11.704812 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:11.705060 (+ 248us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:11.705842 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:11.706750 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:11.707756 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:11.708180 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:11.708296 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:11.710654 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:11.710658 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:11.710667 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:11.710688 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:11.712378 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:11.712873 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:11.712879 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:11.712884 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:11.713048 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:11.713332 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:11.713337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:11.713338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:11.713655 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:11.713859 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:11.714069 (+ 210us) server_negotiation.cc:299] Negotiation successful
0621 07:47:11.714324 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":119,"threads_started":1}
I20260621 07:47:11.715361 1377 sys_catalog.cc:564] T 00000000000000000000000000000000 P e19e8bb02b694d3c9b0ff9209bff085b [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:47:11.716064 1383 sys_catalog.cc:455] T 00000000000000000000000000000000 P e19e8bb02b694d3c9b0ff9209bff085b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e19e8bb02b694d3c9b0ff9209bff085b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19e8bb02b694d3c9b0ff9209bff085b" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45639 } } }
I20260621 07:47:11.716174 1383 sys_catalog.cc:458] T 00000000000000000000000000000000 P e19e8bb02b694d3c9b0ff9209bff085b [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:11.716495 1384 sys_catalog.cc:455] T 00000000000000000000000000000000 P e19e8bb02b694d3c9b0ff9209bff085b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e19e8bb02b694d3c9b0ff9209bff085b. Latest consensus state: current_term: 1 leader_uuid: "e19e8bb02b694d3c9b0ff9209bff085b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19e8bb02b694d3c9b0ff9209bff085b" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45639 } } }
I20260621 07:47:11.716614 1384 sys_catalog.cc:458] T 00000000000000000000000000000000 P e19e8bb02b694d3c9b0ff9209bff085b [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:11.717340 1391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:47:11.719786 1391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:47:11.724412 1391 catalog_manager.cc:1347] Generated new cluster ID: 1fa30351329f438897840a3dbeaaa920
I20260621 07:47:11.724464 1391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:47:11.735221 1391 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:47:11.736063 1391 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:47:11.744724 1391 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e19e8bb02b694d3c9b0ff9209bff085b: Generated new TSK 0
I20260621 07:47:11.745268 1391 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.192.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab.
Entry for principal kudu/127.0.192.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab.
I20260621 07:47:11.773012 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.193:0
--local_ip_for_outbound_sockets=127.0.192.193
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:45639
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:43159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:11.889966 1403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:11.890203 1403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:11.890271 1403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:11.893985 1403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:11.894055 1403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:11.894134 1403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.193
I20260621 07:47:11.898733 1403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:43159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:45639
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.193
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:11.899530 1403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:11.900377 1403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:11.908329 1408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:11.908341 1411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:11.908602 1403 server_base.cc:1034] running on GCE node
W20260621 07:47:11.908649 1409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:11.908938 1403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:11.909437 1403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:11.910605 1403 hybrid_clock.cc:648] HybridClock initialized: now 1782028031910589 us; error 26 us; skew 500 ppm
Jun 21 07:47:11 dist-test-slave-lzx3 krb5kdc[1296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:11.913026 1403 init.cc:377] Logged in from keytab as kudu/127.0.192.193@KRBTEST.COM (short username kudu)
I20260621 07:47:11.913725 1403 webserver.cc:460] Webserver started at http://127.0.192.193:44595/ using document root <none> and password file <none>
I20260621 07:47:11.914204 1403 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:11.914281 1403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:11.914484 1403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:11.916386 1403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-0/data/instance:
uuid: "6edfef8570504a1990c155b62f4d2c02"
format_stamp: "Formatted at 2026-06-21 07:47:11 on dist-test-slave-lzx3"
server_key: "12cde97e76bef4adac554fba3b226efb"
server_key_iv: "cda6e22e35158762e34b2879f239d466"
server_key_version: "encryptionkey@0"
I20260621 07:47:11.916919 1403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-0/wal/instance:
uuid: "6edfef8570504a1990c155b62f4d2c02"
format_stamp: "Formatted at 2026-06-21 07:47:11 on dist-test-slave-lzx3"
server_key: "12cde97e76bef4adac554fba3b226efb"
server_key_iv: "cda6e22e35158762e34b2879f239d466"
server_key_version: "encryptionkey@0"
I20260621 07:47:11.920110 1403 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260621 07:47:11.922228 1418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:11.923303 1403 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 07:47:11.923416 1403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-0/wal
uuid: "6edfef8570504a1990c155b62f4d2c02"
format_stamp: "Formatted at 2026-06-21 07:47:11 on dist-test-slave-lzx3"
server_key: "12cde97e76bef4adac554fba3b226efb"
server_key_iv: "cda6e22e35158762e34b2879f239d466"
server_key_version: "encryptionkey@0"
I20260621 07:47:11.923513 1403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:11.934923 1403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:11.937228 1403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:11.937400 1403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:11.937898 1403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:47:11.938602 1403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:47:11.938668 1403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:11.938724 1403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:47:11.938763 1403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:11.946193 1403 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.193:43359
I20260621 07:47:11.946220 1531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.193:43359 every 8 connection(s)
I20260621 07:47:11.946882 1403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
I20260621 07:47:11.948098 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 1403
I20260621 07:47:11.948220 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-0/wal/instance
I20260621 07:47:11.948489 771 external_mini_cluster.cc:1428] Setting key 38e7c3545c94de87867f6590110844d1
Jun 21 07:47:11 dist-test-slave-lzx3 krb5kdc[1296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.192.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.194@KRBTEST.COM" created.
I20260621 07:47:11.960860 1380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:11.948318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:54833 (local address 127.0.192.254:45639)
0621 07:47:11.948458 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:11.948462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:11.949290 (+ 828us) server_negotiation.cc:407] Connection header received
0621 07:47:11.950829 (+ 1539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:11.950832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:11.950868 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:11.950928 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:11.952110 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:11.953006 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:11.954080 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:11.954336 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:11.954415 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:11.956993 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:11.956997 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:11.957000 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:11.957023 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:11.958559 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:11.959139 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:11.959144 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:11.959146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:11.959231 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:11.959614 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:11.959619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:11.959620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:11.959866 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:11.959978 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:11.960541 (+ 563us) server_negotiation.cc:299] Negotiation successful
0621 07:47:11.960683 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 07:47:11.961561 1534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:11.948694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45639 (local address 127.0.192.193:54833)
0621 07:47:11.949161 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:11.949210 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:11.950627 (+ 1417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:11.951067 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:11.951076 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:11.951507 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:11.951964 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:11.951976 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:11.953115 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:11.953117 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:11.953957 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:11.953966 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:11.954487 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:11.954493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:11.954741 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:11.955269 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:11.955292 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:11.956843 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:11.958757 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:11.958762 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:11.958774 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:11.959014 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:11.959337 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:11.959340 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:11.959343 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:11.959513 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:11.959976 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:11.959982 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:11.960248 (+ 266us) client_negotiation.cc:769] Sending connection context
0621 07:47:11.960471 (+ 223us) client_negotiation.cc:240] Negotiation successful
0621 07:47:11.960793 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":143,"threads_started":1}
I20260621 07:47:11.962731 1532 heartbeater.cc:347] Connected to a master server at 127.0.192.254:45639
I20260621 07:47:11.962985 1532 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:11.963428 1532 heartbeater.cc:511] Master 127.0.192.254:45639 requested a full tablet report, sending...
I20260621 07:47:11.964959 1341 ts_manager.cc:194] Registered new tserver with Master: 6edfef8570504a1990c155b62f4d2c02 (127.0.192.193:43359)
I20260621 07:47:11.966080 1341 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.193@KRBTEST.COM'} at 127.0.192.193:54833
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab.
Entry for principal kudu/127.0.192.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab.
I20260621 07:47:11.974951 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.194:0
--local_ip_for_outbound_sockets=127.0.192.194
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:45639
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:43159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:12.071355 1537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:12.071653 1537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:12.071728 1537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:12.075417 1537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:12.075489 1537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:12.075570 1537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.194
I20260621 07:47:12.080400 1537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:43159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:45639
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.194
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:12.081319 1537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:12.082223 1537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:12.090979 1542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:12.090979 1545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:12.091123 1537 server_base.cc:1034] running on GCE node
W20260621 07:47:12.091289 1543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:12.091720 1537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:12.092242 1537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:12.093398 1537 hybrid_clock.cc:648] HybridClock initialized: now 1782028032093379 us; error 30 us; skew 500 ppm
Jun 21 07:47:12 dist-test-slave-lzx3 krb5kdc[1296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:12.096083 1537 init.cc:377] Logged in from keytab as kudu/127.0.192.194@KRBTEST.COM (short username kudu)
I20260621 07:47:12.096917 1537 webserver.cc:460] Webserver started at http://127.0.192.194:46229/ using document root <none> and password file <none>
I20260621 07:47:12.097390 1537 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:12.097472 1537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:12.097668 1537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:12.099397 1537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-1/data/instance:
uuid: "a69b9c5494ea4227b2fbd60b173afe46"
format_stamp: "Formatted at 2026-06-21 07:47:12 on dist-test-slave-lzx3"
server_key: "9a7a27eb316154edfc91bd44f44ec48b"
server_key_iv: "f6faacf85d21d8f80782d48507797c44"
server_key_version: "encryptionkey@0"
I20260621 07:47:12.099907 1537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-1/wal/instance:
uuid: "a69b9c5494ea4227b2fbd60b173afe46"
format_stamp: "Formatted at 2026-06-21 07:47:12 on dist-test-slave-lzx3"
server_key: "9a7a27eb316154edfc91bd44f44ec48b"
server_key_iv: "f6faacf85d21d8f80782d48507797c44"
server_key_version: "encryptionkey@0"
I20260621 07:47:12.103152 1537 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 07:47:12.105160 1552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:12.106197 1537 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:12.106305 1537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-1/wal
uuid: "a69b9c5494ea4227b2fbd60b173afe46"
format_stamp: "Formatted at 2026-06-21 07:47:12 on dist-test-slave-lzx3"
server_key: "9a7a27eb316154edfc91bd44f44ec48b"
server_key_iv: "f6faacf85d21d8f80782d48507797c44"
server_key_version: "encryptionkey@0"
I20260621 07:47:12.106395 1537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:12.120249 1537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:12.123121 1537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:12.123307 1537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:12.123762 1537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:47:12.124452 1537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:47:12.124516 1537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:12.124591 1537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:47:12.124634 1537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:12.132122 1537 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.194:40831
I20260621 07:47:12.132146 1665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.194:40831 every 8 connection(s)
I20260621 07:47:12.132936 1537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
I20260621 07:47:12.139473 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 1537
I20260621 07:47:12.139597 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-1/wal/instance
I20260621 07:47:12.139834 771 external_mini_cluster.cc:1428] Setting key b0500dc11b4b7ec7d6bb976ede64eea1
Jun 21 07:47:12 dist-test-slave-lzx3 krb5kdc[1296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:47:12.144383 1380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:12.134371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:42271 (local address 127.0.192.254:45639)
0621 07:47:12.134513 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:12.134517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:12.135170 (+ 653us) server_negotiation.cc:407] Connection header received
0621 07:47:12.136234 (+ 1064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:12.136236 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:12.136277 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:12.136342 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:12.137301 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:12.137803 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:12.138735 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:12.138949 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:12.139015 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:12.141303 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:12.141308 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:12.141314 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:12.141333 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:12.142636 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:12.143144 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:12.143149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:12.143150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:12.143219 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:12.143518 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:12.143521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:12.143522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:12.143664 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:12.143754 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:12.144144 (+ 390us) server_negotiation.cc:299] Negotiation successful
0621 07:47:12.144246 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 07:47:12.145087 1668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:12.134669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45639 (local address 127.0.192.194:42271)
0621 07:47:12.135024 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:12.135058 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:12.136069 (+ 1011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:12.136474 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:12.136479 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:12.136794 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:12.137177 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:12.137188 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:12.137896 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:12.137898 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:12.138644 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:12.138651 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:12.139053 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:12.139055 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:12.139171 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:12.139678 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:12.139701 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:12.141133 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:12.142748 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:12.142753 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:12.142765 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:12.143037 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:12.143304 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:12.143307 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:12.143309 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:12.143430 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:12.143754 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:12.143760 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:12.143947 (+ 187us) client_negotiation.cc:769] Sending connection context
0621 07:47:12.144096 (+ 149us) client_negotiation.cc:240] Negotiation successful
0621 07:47:12.144336 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":91,"threads_started":1}
I20260621 07:47:12.146070 1666 heartbeater.cc:347] Connected to a master server at 127.0.192.254:45639
I20260621 07:47:12.146281 1666 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:12.146656 1666 heartbeater.cc:511] Master 127.0.192.254:45639 requested a full tablet report, sending...
I20260621 07:47:12.147612 1341 ts_manager.cc:194] Registered new tserver with Master: a69b9c5494ea4227b2fbd60b173afe46 (127.0.192.194:40831)
WARNING: no policy specified for kudu/127.0.192.195@KRBTEST.COM; defaulting to no policy
I20260621 07:47:12.148351 1341 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.194@KRBTEST.COM'} at 127.0.192.194:42271
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab.
Entry for principal kudu/127.0.192.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab.
I20260621 07:47:12.165118 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.195:0
--local_ip_for_outbound_sockets=127.0.192.195
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:45639
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:43159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:12.262394 1671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:12.262619 1671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:12.262692 1671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:12.266373 1671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:12.266417 1671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:12.266482 1671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.195
I20260621 07:47:12.271173 1671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:43159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:45639
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.195
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:12.272121 1671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:12.273104 1671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:12.282126 1677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:12.282116 1676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:12.282114 1679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:12.282464 1671 server_base.cc:1034] running on GCE node
I20260621 07:47:12.282855 1671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:12.283432 1671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:12.284610 1671 hybrid_clock.cc:648] HybridClock initialized: now 1782028032284582 us; error 39 us; skew 500 ppm
Jun 21 07:47:12 dist-test-slave-lzx3 krb5kdc[1296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:12.286878 1671 init.cc:377] Logged in from keytab as kudu/127.0.192.195@KRBTEST.COM (short username kudu)
I20260621 07:47:12.287683 1671 webserver.cc:460] Webserver started at http://127.0.192.195:44185/ using document root <none> and password file <none>
I20260621 07:47:12.288091 1671 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:12.288138 1671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:12.288264 1671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:12.289863 1671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-2/data/instance:
uuid: "56cdec0a3b234317b3d40d7186ec019c"
format_stamp: "Formatted at 2026-06-21 07:47:12 on dist-test-slave-lzx3"
server_key: "77a22a4a3b1b382685cd3e213d17c02e"
server_key_iv: "618d6b39793a3fd60149278513ecbcae"
server_key_version: "encryptionkey@0"
I20260621 07:47:12.290275 1671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-2/wal/instance:
uuid: "56cdec0a3b234317b3d40d7186ec019c"
format_stamp: "Formatted at 2026-06-21 07:47:12 on dist-test-slave-lzx3"
server_key: "77a22a4a3b1b382685cd3e213d17c02e"
server_key_iv: "618d6b39793a3fd60149278513ecbcae"
server_key_version: "encryptionkey@0"
I20260621 07:47:12.293416 1671 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260621 07:47:12.295253 1686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:12.296233 1671 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:12.296332 1671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-2/wal
uuid: "56cdec0a3b234317b3d40d7186ec019c"
format_stamp: "Formatted at 2026-06-21 07:47:12 on dist-test-slave-lzx3"
server_key: "77a22a4a3b1b382685cd3e213d17c02e"
server_key_iv: "618d6b39793a3fd60149278513ecbcae"
server_key_version: "encryptionkey@0"
I20260621 07:47:12.296413 1671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:12.323232 1671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:12.325407 1671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:12.325567 1671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:12.326108 1671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:47:12.326786 1671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:47:12.326826 1671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:12.326880 1671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:47:12.326912 1671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:12.334522 1671 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.195:43205
I20260621 07:47:12.334544 1799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.195:43205 every 8 connection(s)
I20260621 07:47:12.335218 1671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
I20260621 07:47:12.340029 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 1671
I20260621 07:47:12.340143 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782028028522580-771-0/minicluster-data/ts-2/wal/instance
I20260621 07:47:12.340363 771 external_mini_cluster.cc:1428] Setting key 5d8800601131120cafe7140b173dea04
Jun 21 07:47:12 dist-test-slave-lzx3 krb5kdc[1296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:47:12.347043 1380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:12.336692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:46291 (local address 127.0.192.254:45639)
0621 07:47:12.336818 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:12.336822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:12.337488 (+ 666us) server_negotiation.cc:407] Connection header received
0621 07:47:12.338684 (+ 1196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:12.338688 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:12.338740 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:12.338838 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:12.339813 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:12.340382 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:12.341432 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:12.341636 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:12.341761 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:12.343897 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:12.343900 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:12.343904 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:12.343930 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:12.345252 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:12.345713 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:12.345718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:12.345719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:12.345791 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:12.346117 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:12.346119 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:12.346121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:12.346277 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:12.346382 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:12.346800 (+ 418us) server_negotiation.cc:299] Negotiation successful
0621 07:47:12.346906 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 07:47:12.347685 1802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:12.336974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45639 (local address 127.0.192.195:46291)
0621 07:47:12.337359 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:12.337394 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:12.338497 (+ 1103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:12.338995 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:12.339001 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:12.339282 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:12.339678 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:12.339689 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:12.340517 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:12.340519 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:12.341291 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:12.341299 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:12.341791 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:12.341795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:12.341955 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:12.342472 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:12.342487 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:12.343733 (+ 1246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:12.345370 (+ 1637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:12.345375 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:12.345387 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:12.345619 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:12.345880 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:12.345883 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:12.345885 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:12.346020 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:12.346391 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:12.346399 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:12.346601 (+ 202us) client_negotiation.cc:769] Sending connection context
0621 07:47:12.346753 (+ 152us) client_negotiation.cc:240] Negotiation successful
0621 07:47:12.346999 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":96,"threads_started":1}
I20260621 07:47:12.348625 1800 heartbeater.cc:347] Connected to a master server at 127.0.192.254:45639
I20260621 07:47:12.348819 1800 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:12.349226 1800 heartbeater.cc:511] Master 127.0.192.254:45639 requested a full tablet report, sending...
I20260621 07:47:12.350036 1341 ts_manager.cc:194] Registered new tserver with Master: 56cdec0a3b234317b3d40d7186ec019c (127.0.192.195:43205)
I20260621 07:47:12.350474 1341 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.195@KRBTEST.COM'} at 127.0.192.195:46291
I20260621 07:47:12.353609 771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:47:12.363121 1380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:12.356522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59180 (local address 127.0.192.254:45639)
0621 07:47:12.356682 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:12.356687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:12.356776 (+ 89us) server_negotiation.cc:407] Connection header received
0621 07:47:12.356860 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:12.356862 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:12.356893 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:12.356950 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:12.357463 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:12.357870 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:12.358624 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:12.358795 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:12.358893 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:12.360185 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:12.360188 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:12.360190 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:12.360210 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:12.361626 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:12.361999 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:12.362003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:12.362004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:12.362079 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:12.362311 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:12.362313 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:12.362314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:12.362513 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:12.362616 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:12.362869 (+ 253us) server_negotiation.cc:299] Negotiation successful
0621 07:47:12.362954 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260621 07:47:12.367242 1341 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:47:12.369987 1341 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:47:12.382341 1811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:12.377097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:40831 (local address 127.0.0.1:38924)
0621 07:47:12.377535 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:12.377576 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:12.377671 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:12.378643 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:12.378648 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:12.378669 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:12.378769 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:12.378775 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:12.379788 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:12.379794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:12.380913 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:12.380922 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:12.381886 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:12.381890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:12.381983 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:12.382006 (+ 23us) client_negotiation.cc:769] Sending connection context
0621 07:47:12.382070 (+ 64us) client_negotiation.cc:240] Negotiation successful
0621 07:47:12.382187 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":1,"thread_start_us":46,"threads_started":1}
I20260621 07:47:12.382463 1816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:12.378032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35396 (local address 127.0.192.193:43359)
0621 07:47:12.378360 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:12.378371 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:12.378391 (+ 20us) server_negotiation.cc:407] Connection header received
0621 07:47:12.378460 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:12.378464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:12.378641 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:12.379040 (+ 399us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:12.379178 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:12.380137 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:12.381170 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:12.381713 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:12.381863 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:12.381997 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:12.382190 (+ 193us) server_negotiation.cc:299] Negotiation successful
0621 07:47:12.382301 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":120,"threads_started":1}
I20260621 07:47:12.382674 1812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:12.377251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.195:43205 (local address 127.0.0.1:56098)
0621 07:47:12.377927 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:12.377940 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:12.378020 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:12.378753 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:12.378755 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:12.378767 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:12.378856 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:12.378861 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:12.380160 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:12.380163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:12.381174 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:12.381181 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:12.382094 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:12.382097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:12.382191 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:12.382212 (+ 21us) client_negotiation.cc:769] Sending connection context
0621 07:47:12.382287 (+ 75us) client_negotiation.cc:240] Negotiation successful
0621 07:47:12.382343 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":46,"thread_start_us":87,"threads_started":1}
I20260621 07:47:12.382849 1814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:12.377656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:43359 (local address 127.0.0.1:35396)
0621 07:47:12.378033 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:12.378045 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:12.378110 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:12.378781 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:12.378786 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:12.378807 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:12.378885 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:12.378889 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:12.380255 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:12.380259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:12.381056 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:12.381063 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:12.381853 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:12.381857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:12.381957 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:12.381991 (+ 34us) client_negotiation.cc:769] Sending connection context
0621 07:47:12.382094 (+ 103us) client_negotiation.cc:240] Negotiation successful
0621 07:47:12.382651 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":479,"thread_start_us":88,"threads_started":1}
I20260621 07:47:12.382855 1815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:12.377671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56098 (local address 127.0.192.195:43205)
0621 07:47:12.378058 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:12.378069 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:12.378087 (+ 18us) server_negotiation.cc:407] Connection header received
0621 07:47:12.378158 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:12.378165 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:12.378487 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:12.378624 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:12.378963 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:12.380033 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:12.381280 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:12.381992 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:12.382096 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:12.382216 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:12.382311 (+ 95us) server_negotiation.cc:299] Negotiation successful
0621 07:47:12.382496 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":129,"threads_started":1}
I20260621 07:47:12.383183 1813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:12.377228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38924 (local address 127.0.192.194:40831)
0621 07:47:12.377851 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:12.377861 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:12.377890 (+ 29us) server_negotiation.cc:407] Connection header received
0621 07:47:12.378344 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:12.378349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:12.378484 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:12.378903 (+ 419us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:12.379043 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:12.379662 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:12.381032 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:12.381746 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:12.382765 (+ 1019us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:12.382906 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:12.382967 (+ 61us) server_negotiation.cc:299] Negotiation successful
0621 07:47:12.383043 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":123,"threads_started":1}
I20260621 07:47:12.385493 1600 tablet_service.cc:1467] Processing CreateTablet for tablet 619b782f21704e858be0e75e546885ea (DEFAULT_TABLE table=test-table [id=121730f27cd848ea8ae0b94b1be73704]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:47:12.385598 1734 tablet_service.cc:1467] Processing CreateTablet for tablet 619b782f21704e858be0e75e546885ea (DEFAULT_TABLE table=test-table [id=121730f27cd848ea8ae0b94b1be73704]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:47:12.385537 1466 tablet_service.cc:1467] Processing CreateTablet for tablet 619b782f21704e858be0e75e546885ea (DEFAULT_TABLE table=test-table [id=121730f27cd848ea8ae0b94b1be73704]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:47:12.386329 1600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 619b782f21704e858be0e75e546885ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:12.386390 1466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 619b782f21704e858be0e75e546885ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:12.386390 1734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 619b782f21704e858be0e75e546885ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:12.391289 1817 tablet_bootstrap.cc:492] T 619b782f21704e858be0e75e546885ea P a69b9c5494ea4227b2fbd60b173afe46: Bootstrap starting.
I20260621 07:47:12.393170 1819 tablet_bootstrap.cc:492] T 619b782f21704e858be0e75e546885ea P 56cdec0a3b234317b3d40d7186ec019c: Bootstrap starting.
I20260621 07:47:12.393170 1818 tablet_bootstrap.cc:492] T 619b782f21704e858be0e75e546885ea P 6edfef8570504a1990c155b62f4d2c02: Bootstrap starting.
I20260621 07:47:12.394076 1817 tablet_bootstrap.cc:654] T 619b782f21704e858be0e75e546885ea P a69b9c5494ea4227b2fbd60b173afe46: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:12.394686 1817 log.cc:824] T 619b782f21704e858be0e75e546885ea P a69b9c5494ea4227b2fbd60b173afe46: Log is configured to *not* fsync() on all Append() calls
I20260621 07:47:12.394942 1818 tablet_bootstrap.cc:654] T 619b782f21704e858be0e75e546885ea P 6edfef8570504a1990c155b62f4d2c02: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:12.394948 1819 tablet_bootstrap.cc:654] T 619b782f21704e858be0e75e546885ea P 56cdec0a3b234317b3d40d7186ec019c: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:12.396047 1819 log.cc:824] T 619b782f21704e858be0e75e546885ea P 56cdec0a3b234317b3d40d7186ec019c: Log is configured to *not* fsync() on all Append() calls
I20260621 07:47:12.396047 1818 log.cc:824] T 619b782f21704e858be0e75e546885ea P 6edfef8570504a1990c155b62f4d2c02: Log is configured to *not* fsync() on all Append() calls
I20260621 07:47:12.396219 1817 tablet_bootstrap.cc:492] T 619b782f21704e858be0e75e546885ea P a69b9c5494ea4227b2fbd60b173afe46: No bootstrap required, opened a new log
I20260621 07:47:12.396395 1817 ts_tablet_manager.cc:1397] T 619b782f21704e858be0e75e546885ea P a69b9c5494ea4227b2fbd60b173afe46: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260621 07:47:12.397680 1818 tablet_bootstrap.cc:492] T 619b782f21704e858be0e75e546885ea P 6edfef8570504a1990c155b62f4d2c02: No bootstrap required, opened a new log
I20260621 07:47:12.397722 1819 tablet_bootstrap.cc:492] T 619b782f21704e858be0e75e546885ea P 56cdec0a3b234317b3d40d7186ec019c: No bootstrap required, opened a new log
I20260621 07:47:12.397861 1819 ts_tablet_manager.cc:1397] T 619b782f21704e858be0e75e546885ea P 56cdec0a3b234317b3d40d7186ec019c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260621 07:47:12.397861 1818 ts_tablet_manager.cc:1397] T 619b782f21704e858be0e75e546885ea P 6edfef8570504a1990c155b62f4d2c02: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260621 07:47:12.399747 1817 raft_consensus.cc:348] T 619b782f21704e858be0e75e546885ea P a69b9c5494ea4227b2fbd60b173afe46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a69b9c5494ea4227b2fbd60b173afe46" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40831 } } peers { permanent_uuid: "56cdec0a3b234317b3d40d7186ec019c" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43205 } } peers { permanent_uuid: "6edfef8570504a1990c155b62f4d2c02" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 43359 } }
I20260621 07:47:12.399986 1817 raft_consensus.cc:374] T 619b782f21704e858be0e75e546885ea P a69b9c5494ea4227b2fbd60b173afe46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:12.400038 1817 raft_consensus.cc:729] T 619b782f21704e858be0e75e546885ea P a69b9c5494ea4227b2fbd60b173afe46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a69b9c5494ea4227b2fbd60b173afe46, State: Initialized, Role: FOLLOWER
I20260621 07:47:12.400296 1818 raft_consensus.cc:348] T 619b782f21704e858be0e75e546885ea P 6edfef8570504a1990c155b62f4d2c02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a69b9c5494ea4227b2fbd60b173afe46" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40831 } } peers { permanent_uuid: "56cdec0a3b234317b3d40d7186ec019c" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43205 } } peers { permanent_uuid: "6edfef8570504a1990c155b62f4d2c02" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 43359 } }
I20260621 07:47:12.400405 1817 consensus_queue.cc:260] T 619b782f21704e858be0e75e546885ea P a69b9c5494ea4227b2fbd60b173afe46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a69b9c5494ea4227b2fbd60b173afe46" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40831 } } peers { permanent_uuid: "56cdec0a3b234317b3d40d7186ec019c" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43205 } } peers { permanent_uuid: "6edfef8570504a1990c155b62f4d2c02" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 43359 } }
I20260621 07:47:12.400513 1818 raft_consensus.cc:374] T 619b782f21704e858be0e75e546885ea P 6edfef8570504a1990c155b62f4d2c02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:12.400557 1818 raft_consensus.cc:729] T 619b782f21704e858be0e75e546885ea P 6edfef8570504a1990c155b62f4d2c02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edfef8570504a1990c155b62f4d2c02, State: Initialized, Role: FOLLOWER
I20260621 07:47:12.401012 1666 heartbeater.cc:503] Master 127.0.192.254:45639 was elected leader, sending a full tablet report...
I20260621 07:47:12.400977 1818 consensus_queue.cc:260] T 619b782f21704e858be0e75e546885ea P 6edfef8570504a1990c155b62f4d2c02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a69b9c5494ea4227b2fbd60b173afe46" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40831 } } peers { permanent_uuid: "56cdec0a3b234317b3d40d7186ec019c" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43205 } } peers { permanent_uuid: "6edfef8570504a1990c155b62f4d2c02" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 43359 } }
I20260621 07:47:12.401271 1817 ts_tablet_manager.cc:1428] T 619b782f21704e858be0e75e546885ea P a69b9c5494ea4227b2fbd60b173afe46: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260621 07:47:12.401594 1532 heartbeater.cc:503] Master 127.0.192.254:45639 was elected leader, sending a full tablet report...
I20260621 07:47:12.401873 1818 ts_tablet_manager.cc:1428] T 619b782f21704e858be0e75e546885ea P 6edfef8570504a1990c155b62f4d2c02: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260621 07:47:12.402067 1819 raft_consensus.cc:348] T 619b782f21704e858be0e75e546885ea P 56cdec0a3b234317b3d40d7186ec019c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a69b9c5494ea4227b2fbd60b173afe46" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40831 } } peers { permanent_uuid: "56cdec0a3b234317b3d40d7186ec019c" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43205 } } peers { permanent_uuid: "6edfef8570504a1990c155b62f4d2c02" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 43359 } }
I20260621 07:47:12.402297 1819 raft_consensus.cc:374] T 619b782f21704e858be0e75e546885ea P 56cdec0a3b234317b3d40d7186ec019c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:12.402339 1819 raft_consensus.cc:729] T 619b782f21704e858be0e75e546885ea P 56cdec0a3b234317b3d40d7186ec019c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56cdec0a3b234317b3d40d7186ec019c, State: Initialized, Role: FOLLOWER
I20260621 07:47:12.402729 1819 consensus_queue.cc:260] T 619b782f21704e858be0e75e546885ea P 56cdec0a3b234317b3d40d7186ec019c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a69b9c5494ea4227b2fbd60b173afe46" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40831 } } peers { permanent_uuid: "56cdec0a3b234317b3d40d7186ec019c" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43205 } } peers { permanent_uuid: "6edfef8570504a1990c155b62f4d2c02" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 43359 } }
I20260621 07:47:12.403332 1800 heartbeater.cc:503] Master 127.0.192.254:45639 was elected leader, sending a full tablet report...
I20260621 07:47:12.403439 1819 ts_tablet_manager.cc:1428] T 619b782f21704e858be0e75e546885ea P 56cdec0a3b234317b3d40d7186ec019c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260621 07:47:12.442930 1825 raft_consensus.cc:482] T 619b782f21704e858be0e75e546885ea P 56cdec0a3b234317b3d40d7186ec019c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:47:12.443071 1825 raft_consensus.cc:504] T 619b782f21704e858be0e75e546885ea P 56cdec0a3b234317b3d40d7186ec019c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a69b9c5494ea4227b2fbd60b173afe46" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40831 } } peers { permanent_uuid: "56cdec0a3b234317b3d40d7186ec019c" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43205 } } peers { permanent_uuid: "6edfef8570504a1990c155b62f4d2c02" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 43359 } }
I20260621 07:47:12.443874 1825 leader_election.cc:288] T 619b782f21704e858be0e75e546885ea P 56cdec0a3b234317b3d40d7186ec019c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a69b9c5494ea4227b2fbd60b173afe46 (127.0.192.194:40831), 6edfef8570504a1990c155b62f4d2c02 (127.0.192.193:43359)
W20260621 07:47:12.447754 1533 tablet.cc:2369] T 619b782f21704e858be0e75e546885ea P 6edfef8570504a1990c155b62f4d2c02: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:47:12.448299 1826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:12.444288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:43359 (local address 127.0.192.195:45769)
0621 07:47:12.444528 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:12.444539 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:12.444636 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:12.444863 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:12.444865 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:12.444877 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:12.444958 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:12.444961 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:12.445738 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:12.445740 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:12.446750 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:12.446757 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:12.447931 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:12.447934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:12.448017 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:12.448039 (+ 22us) client_negotiation.cc:769] Sending connection context
0621 07:47:12.448111 (+ 72us) client_negotiation.cc:240] Negotiation successful
0621 07:47:12.448170 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":87,"threads_started":1}
I20260621 07:47:12.448330 1816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:12.444474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:45769 (local address 127.0.192.193:43359)
0621 07:47:12.444603 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:12.444606 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:12.444624 (+ 18us) server_negotiation.cc:407] Connection header received
0621 07:47:12.444738 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:12.444743 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:12.444784 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:12.444850 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:12.445060 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:12.445662 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:12.446845 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:12.447772 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:12.447882 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:12.447924 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:12.448143 (+ 219us) server_negotiation.cc:299] Negotiation successful
0621 07:47:12.448210 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260621 07:47:12.448417 1802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:12.444054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:40831 (local address 127.0.192.195:37967)
0621 07:47:12.444154 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:12.444170 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:12.444243 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:12.444503 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:12.444507 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:12.444522 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:12.444653 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:12.444662 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:12.445577 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:12.445578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:12.446705 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:12.446713 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:12.447979 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:12.447982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:12.448073 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:12.448112 (+ 39us) client_negotiation.cc:769] Sending connection context
0621 07:47:12.448178 (+ 66us) client_negotiation.cc:240] Negotiation successful
0621 07:47:12.448234 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":49}
I20260621 07:47:12.448529 1813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:12.444176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:37967 (local address 127.0.192.194:40831)
0621 07:47:12.444281 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:12.444285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:12.444297 (+ 12us) server_negotiation.cc:407] Connection header received
0621 07:47:12.444347 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:12.444349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:12.444405 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:12.444480 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:12.444758 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:12.445489 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:12.446835 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:12.447793 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:12.447888 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:12.447924 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:12.448345 (+ 421us) server_negotiation.cc:299] Negotiation successful
0621 07:47:12.448415 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260621 07:47:12.449163 1620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "619b782f21704e858be0e75e546885ea" candidate_uuid: "56cdec0a3b234317b3d40d7186ec019c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a69b9c5494ea4227b2fbd60b173afe46" is_pre_election: true
I20260621 07:47:12.449163 1486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "619b782f21704e858be0e75e546885ea" candidate_uuid: "56cdec0a3b234317b3d40d7186ec019c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edfef8570504a1990c155b62f4d2c02" is_pre_election: true
I20260621 07:47:12.449396 1620 raft_consensus.cc:2454] T 619b782f21704e858be0e75e546885ea P a69b9c5494ea4227b2fbd60b173afe46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56cdec0a3b234317b3d40d7186ec019c in term 0.
I20260621 07:47:12.449396 1486 raft_consensus.cc:2454] T 619b782f21704e858be0e75e546885ea P 6edfef8570504a1990c155b62f4d2c02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56cdec0a3b234317b3d40d7186ec019c in term 0.
I20260621 07:47:12.449782 1687 leader_election.cc:302] T 619b782f21704e858be0e75e546885ea P 56cdec0a3b234317b3d40d7186ec019c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56cdec0a3b234317b3d40d7186ec019c, 6edfef8570504a1990c155b62f4d2c02; no voters:
I20260621 07:47:12.449961 1825 raft_consensus.cc:2781] T 619b782f21704e858be0e75e546885ea P 56cdec0a3b234317b3d40d7186ec019c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:47:12.450034 1825 raft_consensus.cc:482] T 619b782f21704e858be0e75e546885ea P 56cdec0a3b234317b3d40d7186ec019c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:47:12.450062 1825 raft_consensus.cc:3037] T 619b782f21704e858be0e75e546885ea P 56cdec0a3b234317b3d40d7186ec019c [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:12.450989 1825 raft_consensus.cc:504] T 619b782f21704e858be0e75e546885ea P 56cdec0a3b234317b3d40d7186ec019c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a69b9c5494ea4227b2fbd60b173afe46" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40831 } } peers { permanent_uuid: "56cdec0a3b234317b3d40d7186ec019c" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43205 } } peers { permanent_uuid: "6edfef8570504a1990c155b62f4d2c02" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 43359 } }
I20260621 07:47:12.451273 1825 leader_election.cc:288] T 619b782f21704e858be0e75e546885ea P 56cdec0a3b234317b3d40d7186ec019c [CANDIDATE]: Term 1 election: Requested vote from peers a69b9c5494ea4227b2fbd60b173afe46 (127.0.192.194:40831), 6edfef8570504a1990c155b62f4d2c02 (127.0.192.193:43359)
I20260621 07:47:12.451627 1620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "619b782f21704e858be0e75e546885ea" candidate_uuid: "56cdec0a3b234317b3d40d7186ec019c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a69b9c5494ea4227b2fbd60b173afe46"
I20260621 07:47:12.451658 1486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "619b782f21704e858be0e75e546885ea" candidate_uuid: "56cdec0a3b234317b3d40d7186ec019c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edfef8570504a1990c155b62f4d2c02"
I20260621 07:47:12.451745 1620 raft_consensus.cc:3037] T 619b782f21704e858be0e75e546885ea P a69b9c5494ea4227b2fbd60b173afe46 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:12.451785 1486 raft_consensus.cc:3037] T 619b782f21704e858be0e75e546885ea P 6edfef8570504a1990c155b62f4d2c02 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:12.452765 1486 raft_consensus.cc:2454] T 619b782f21704e858be0e75e546885ea P 6edfef8570504a1990c155b62f4d2c02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56cdec0a3b234317b3d40d7186ec019c in term 1.
I20260621 07:47:12.452765 1620 raft_consensus.cc:2454] T 619b782f21704e858be0e75e546885ea P a69b9c5494ea4227b2fbd60b173afe46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56cdec0a3b234317b3d40d7186ec019c in term 1.
I20260621 07:47:12.453055 1687 leader_election.cc:302] T 619b782f21704e858be0e75e546885ea P 56cdec0a3b234317b3d40d7186ec019c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56cdec0a3b234317b3d40d7186ec019c, a69b9c5494ea4227b2fbd60b173afe46; no voters:
I20260621 07:47:12.453207 1825 raft_consensus.cc:2781] T 619b782f21704e858be0e75e546885ea P 56cdec0a3b234317b3d40d7186ec019c [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:47:12.453374 1825 raft_consensus.cc:686] T 619b782f21704e858be0e75e546885ea P 56cdec0a3b234317b3d40d7186ec019c [term 1 LEADER]: Becoming Leader. State: Replica: 56cdec0a3b234317b3d40d7186ec019c, State: Running, Role: LEADER
I20260621 07:47:12.453624 1825 consensus_queue.cc:237] T 619b782f21704e858be0e75e546885ea P 56cdec0a3b234317b3d40d7186ec019c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a69b9c5494ea4227b2fbd60b173afe46" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40831 } } peers { permanent_uuid: "56cdec0a3b234317b3d40d7186ec019c" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43205 } } peers { permanent_uuid: "6edfef8570504a1990c155b62f4d2c02" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 43359 } }
I20260621 07:47:12.456270 1340 catalog_manager.cc:5495] T 619b782f21704e858be0e75e546885ea P 56cdec0a3b234317b3d40d7186ec019c reported cstate change: term changed from 0 to 1, leader changed from <none> to 56cdec0a3b234317b3d40d7186ec019c (127.0.192.195). New cstate: current_term: 1 leader_uuid: "56cdec0a3b234317b3d40d7186ec019c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a69b9c5494ea4227b2fbd60b173afe46" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56cdec0a3b234317b3d40d7186ec019c" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6edfef8570504a1990c155b62f4d2c02" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 43359 } health_report { overall_health: UNKNOWN } } }
Jun 21 07:47:12 dist-test-slave-lzx3 krb5kdc[1296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.192.193@KRBTEST.COM
I20260621 07:47:12.483777 1816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:12.475775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35410 (local address 127.0.192.193:43359)
0621 07:47:12.475889 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:12.475894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:12.476077 (+ 183us) server_negotiation.cc:407] Connection header received
0621 07:47:12.476226 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:12.476229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:12.476276 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:12.476366 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:12.477066 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:12.477533 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:12.478278 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:12.478472 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:12.478568 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:12.480825 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:12.480835 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:12.480845 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:12.480867 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:12.482481 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:12.482828 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:12.482832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:12.482833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:12.482924 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:12.483139 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:12.483141 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:12.483142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:12.483332 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:12.483472 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:12.483603 (+ 131us) server_negotiation.cc:299] Negotiation successful
0621 07:47:12.483678 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Jun 21 07:47:12 dist-test-slave-lzx3 krb5kdc[1296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028032, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 07:47:12 dist-test-slave-lzx3 krb5kdc[1296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028032, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.192.193@KRBTEST.COM
I20260621 07:47:12.506249 1816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:12.499271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35414 (local address 127.0.192.193:43359)
0621 07:47:12.499375 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:12.499379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:12.499473 (+ 94us) server_negotiation.cc:407] Connection header received
0621 07:47:12.499599 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:12.499602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:12.499656 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:12.499730 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:12.500300 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:12.500795 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:12.501519 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:12.501715 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:12.501833 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:12.503844 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:12.503847 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:12.503848 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:12.503867 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:12.505009 (+ 1142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:12.505330 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:12.505332 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:12.505333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:12.505389 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:12.505591 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:12.505592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:12.505593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:12.505711 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:12.505793 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:12.506085 (+ 292us) server_negotiation.cc:299] Negotiation successful
0621 07:47:12.506157 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Jun 21 07:47:12 dist-test-slave-lzx3 krb5kdc[1296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 07:47:12 dist-test-slave-lzx3 krb5kdc[1296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.192.193@KRBTEST.COM
I20260621 07:47:12.526924 1816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:12.519942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35418 (local address 127.0.192.193:43359)
0621 07:47:12.520060 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:12.520064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:12.520201 (+ 137us) server_negotiation.cc:407] Connection header received
0621 07:47:12.520329 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:12.520332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:12.520370 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:12.520440 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:12.521050 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:12.521481 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:12.522275 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:12.522485 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:12.522589 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:12.524550 (+ 1961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:12.524593 (+ 43us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:12.524598 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:12.524624 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:12.525638 (+ 1014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:12.526028 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:12.526030 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:12.526031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:12.526088 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:12.526348 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:12.526350 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:12.526351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:12.526465 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:12.526545 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:12.526728 (+ 183us) server_negotiation.cc:299] Negotiation successful
0621 07:47:12.526794 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260621 07:47:12.527987 1446 tablet_service.cc:2908] Rejecting scan request for tablet 619b782f21704e858be0e75e546885ea: Uninitialized: clean time has not yet been initialized
I20260621 07:47:12.529714 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 1403
I20260621 07:47:12.535535 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 1537
I20260621 07:47:12.540366 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 1671
I20260621 07:47:12.545239 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 1309
2026-06-21T07:47:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2071 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:47:12 dist-test-slave-lzx3 krb5kdc[1841](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:47:12 dist-test-slave-lzx3 krb5kdc[1841](info): set up 2 sockets
Jun 21 07:47:12 dist-test-slave-lzx3 krb5kdc[1841](info): commencing operation
krb5kdc: starting...
W20260621 07:47:13.536079 771 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/krb5kdc/test-user.keytab.
Jun 21 07:47:13 dist-test-slave-lzx3 krb5kdc[1841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:47:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:47:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.192.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
I20260621 07:47:13.687476 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:44477
--webserver_interface=127.0.192.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:39665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:44477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:13.784224 1854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:13.784415 1854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:13.784464 1854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:13.790073 1854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:47:13.790129 1854 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260621 07:47:13.790153 1854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:13.790189 1854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:13.790225 1854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:47:13.790254 1854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:47:13.797775 1854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:39665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:44477
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:44477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:13.798581 1854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:13.799739 1854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:13.807358 1862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:13.807351 1860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:13.807372 1859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:13.807554 1854 server_base.cc:1034] running on GCE node
I20260621 07:47:13.808138 1854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:13.809093 1854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:13.810266 1854 hybrid_clock.cc:648] HybridClock initialized: now 1782028033810256 us; error 28 us; skew 500 ppm
Jun 21 07:47:13 dist-test-slave-lzx3 krb5kdc[1841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:13.812777 1854 init.cc:377] Logged in from keytab as kudu/127.0.192.254@KRBTEST.COM (short username kudu)
I20260621 07:47:13.813531 1854 webserver.cc:460] Webserver started at http://127.0.192.254:33559/ using document root <none> and password file <none>
I20260621 07:47:13.813987 1854 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:13.814038 1854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:13.814208 1854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:13.815841 1854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "c4822e5a59714dac88935d524bd2299a"
format_stamp: "Formatted at 2026-06-21 07:47:13 on dist-test-slave-lzx3"
server_key: "001d22c77378e596f583666bc45e6b8f"
server_key_iv: "aef18c9713dfe8ed4fa0787b56788ba8"
server_key_version: "encryptionkey@0"
I20260621 07:47:13.816308 1854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "c4822e5a59714dac88935d524bd2299a"
format_stamp: "Formatted at 2026-06-21 07:47:13 on dist-test-slave-lzx3"
server_key: "001d22c77378e596f583666bc45e6b8f"
server_key_iv: "aef18c9713dfe8ed4fa0787b56788ba8"
server_key_version: "encryptionkey@0"
I20260621 07:47:13.819355 1854 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260621 07:47:13.821408 1869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:13.822434 1854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:13.822548 1854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "c4822e5a59714dac88935d524bd2299a"
format_stamp: "Formatted at 2026-06-21 07:47:13 on dist-test-slave-lzx3"
server_key: "001d22c77378e596f583666bc45e6b8f"
server_key_iv: "aef18c9713dfe8ed4fa0787b56788ba8"
server_key_version: "encryptionkey@0"
I20260621 07:47:13.822642 1854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:13.837306 1854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:13.839552 1854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:13.839725 1854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:13.846148 1854 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:44477
I20260621 07:47:13.846153 1931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:44477 every 8 connection(s)
I20260621 07:47:13.846866 1854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:47:13.849124 1932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:13.852684 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 1854
I20260621 07:47:13.852795 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:47:13.853046 771 external_mini_cluster.cc:1428] Setting key 2a3708ed5952cfbcdfa94c41ee7441a5
I20260621 07:47:13.854162 1932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4822e5a59714dac88935d524bd2299a: Bootstrap starting.
I20260621 07:47:13.856194 1932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4822e5a59714dac88935d524bd2299a: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:13.856957 1932 log.cc:824] T 00000000000000000000000000000000 P c4822e5a59714dac88935d524bd2299a: Log is configured to *not* fsync() on all Append() calls
I20260621 07:47:13.858472 1932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4822e5a59714dac88935d524bd2299a: No bootstrap required, opened a new log
Jun 21 07:47:13 dist-test-slave-lzx3 krb5kdc[1841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:47:13.861670 1932 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4822e5a59714dac88935d524bd2299a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4822e5a59714dac88935d524bd2299a" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 44477 } }
I20260621 07:47:13.861917 1932 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4822e5a59714dac88935d524bd2299a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:13.862021 1932 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4822e5a59714dac88935d524bd2299a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4822e5a59714dac88935d524bd2299a, State: Initialized, Role: FOLLOWER
I20260621 07:47:13.862442 1932 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4822e5a59714dac88935d524bd2299a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4822e5a59714dac88935d524bd2299a" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 44477 } }
I20260621 07:47:13.862557 1932 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4822e5a59714dac88935d524bd2299a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:47:13.862640 1932 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4822e5a59714dac88935d524bd2299a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:47:13.862753 1932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4822e5a59714dac88935d524bd2299a [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:13.863515 1932 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4822e5a59714dac88935d524bd2299a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4822e5a59714dac88935d524bd2299a" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 44477 } }
I20260621 07:47:13.863821 1932 leader_election.cc:302] T 00000000000000000000000000000000 P c4822e5a59714dac88935d524bd2299a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4822e5a59714dac88935d524bd2299a; no voters:
I20260621 07:47:13.864087 1932 leader_election.cc:288] T 00000000000000000000000000000000 P c4822e5a59714dac88935d524bd2299a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:47:13.864179 1937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4822e5a59714dac88935d524bd2299a [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:47:13.864374 1937 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4822e5a59714dac88935d524bd2299a [term 1 LEADER]: Becoming Leader. State: Replica: c4822e5a59714dac88935d524bd2299a, State: Running, Role: LEADER
I20260621 07:47:13.864636 1937 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4822e5a59714dac88935d524bd2299a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4822e5a59714dac88935d524bd2299a" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 44477 } }
I20260621 07:47:13.865152 1932 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4822e5a59714dac88935d524bd2299a [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:47:13.865576 1935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:13.854199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56992 (local address 127.0.192.254:44477)
0621 07:47:13.854579 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:13.854592 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:13.854621 (+ 29us) server_negotiation.cc:407] Connection header received
0621 07:47:13.855164 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:13.855178 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:13.855437 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:13.855675 (+ 238us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:13.856491 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:13.857418 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:13.858212 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:13.858703 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:13.858840 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:13.860955 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:13.860960 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:13.860969 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:13.860990 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:13.862889 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:13.863320 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:13.863326 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:13.863330 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:13.863439 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:13.863742 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:13.863745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:13.863747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:13.864169 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:13.864355 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:13.864601 (+ 246us) server_negotiation.cc:299] Negotiation successful
0621 07:47:13.864835 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":109,"threads_started":1}
I20260621 07:47:13.865587 1937 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4822e5a59714dac88935d524bd2299a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4822e5a59714dac88935d524bd2299a. Latest consensus state: current_term: 1 leader_uuid: "c4822e5a59714dac88935d524bd2299a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4822e5a59714dac88935d524bd2299a" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 44477 } } }
I20260621 07:47:13.865705 1937 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4822e5a59714dac88935d524bd2299a [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:13.865949 1938 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4822e5a59714dac88935d524bd2299a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4822e5a59714dac88935d524bd2299a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4822e5a59714dac88935d524bd2299a" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 44477 } } }
I20260621 07:47:13.866039 1938 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4822e5a59714dac88935d524bd2299a [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:13.866343 1946 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:47:13.868911 1946 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:47:13.873677 1946 catalog_manager.cc:1347] Generated new cluster ID: ee6e6de2cd7544d084337645fea31ed8
I20260621 07:47:13.873737 1946 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:47:13.894032 1946 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:47:13.894737 1946 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:47:13.904841 1946 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4822e5a59714dac88935d524bd2299a: Generated new TSK 0
I20260621 07:47:13.905347 1946 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.192.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab.
Entry for principal kudu/127.0.192.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab.
I20260621 07:47:13.936705 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.193:0
--local_ip_for_outbound_sockets=127.0.192.193
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:44477
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:39665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:14.033349 1958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:14.033565 1958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:14.033601 1958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:14.037156 1958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:14.037202 1958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:14.037266 1958 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 07:47:14.037350 1958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.193
I20260621 07:47:14.041963 1958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:39665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.192.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.192.254:44477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.193
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:14.042833 1958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:14.043718 1958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:14.052104 1966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:14.052150 1964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:14.052181 1963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:14.052376 1958 server_base.cc:1034] running on GCE node
I20260621 07:47:14.052781 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:14.053344 1958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:14.054531 1958 hybrid_clock.cc:648] HybridClock initialized: now 1782028034054504 us; error 41 us; skew 500 ppm
Jun 21 07:47:14 dist-test-slave-lzx3 krb5kdc[1841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:14.057090 1958 init.cc:377] Logged in from keytab as kudu/127.0.192.193@KRBTEST.COM (short username kudu)
I20260621 07:47:14.057802 1958 webserver.cc:460] Webserver started at http://127.0.192.193:36253/ using document root <none> and password file <none>
I20260621 07:47:14.058286 1958 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:14.058339 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:14.058558 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:14.060353 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-0/data/instance:
uuid: "dc15feffe082457c99a95478676e263d"
format_stamp: "Formatted at 2026-06-21 07:47:14 on dist-test-slave-lzx3"
server_key: "1b60a8e29bb9252d4a91d3e8369aefc4"
server_key_iv: "f8cf15db85a0a165244a197b69c5b77c"
server_key_version: "encryptionkey@0"
I20260621 07:47:14.060926 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-0/wal/instance:
uuid: "dc15feffe082457c99a95478676e263d"
format_stamp: "Formatted at 2026-06-21 07:47:14 on dist-test-slave-lzx3"
server_key: "1b60a8e29bb9252d4a91d3e8369aefc4"
server_key_iv: "f8cf15db85a0a165244a197b69c5b77c"
server_key_version: "encryptionkey@0"
I20260621 07:47:14.064337 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:47:14.066624 1973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:14.067883 1958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:14.067984 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-0/wal
uuid: "dc15feffe082457c99a95478676e263d"
format_stamp: "Formatted at 2026-06-21 07:47:14 on dist-test-slave-lzx3"
server_key: "1b60a8e29bb9252d4a91d3e8369aefc4"
server_key_iv: "f8cf15db85a0a165244a197b69c5b77c"
server_key_version: "encryptionkey@0"
I20260621 07:47:14.068115 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:14.103183 1958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:14.105885 1958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:14.106057 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:14.107227 1958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:47:14.107295 1958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:14.107358 1958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:47:14.107398 1958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 07:47:14 dist-test-slave-lzx3 krb5kdc[1841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:47:14.117693 1958 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.193:46215
I20260621 07:47:14.117769 2088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.193:46215 every 8 connection(s)
I20260621 07:47:14.118789 1958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
I20260621 07:47:14.121030 1935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:14.108093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:47799 (local address 127.0.192.254:44477)
0621 07:47:14.108205 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:14.108208 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:14.109130 (+ 922us) server_negotiation.cc:407] Connection header received
0621 07:47:14.110017 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:14.110020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:14.110085 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:14.110148 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:14.111711 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.112532 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.114354 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.114775 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.114899 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:14.117786 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:14.117790 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:14.117793 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:14.117816 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:14.119420 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:14.119873 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:14.119876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:14.119877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:14.119943 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:14.120194 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:14.120196 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:14.120197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:14.120338 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:14.120410 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:14.120831 (+ 421us) server_negotiation.cc:299] Negotiation successful
0621 07:47:14.120923 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260621 07:47:14.121666 1982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:14.108508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:44477 (local address 127.0.192.193:47799)
0621 07:47:14.108953 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:14.109020 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:14.109757 (+ 737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:14.110554 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:14.110561 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:14.110986 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:14.111572 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:14.111591 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.112701 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.112709 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:14.114201 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:14.114214 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.114926 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.114931 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:14.115131 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:14.115736 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:14.115760 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:14.117614 (+ 1854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:14.119518 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:14.119522 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:14.119533 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:14.119793 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:14.120016 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:14.120019 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:14.120020 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:14.120125 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:14.120406 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:14.120410 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:14.120652 (+ 242us) client_negotiation.cc:769] Sending connection context
0621 07:47:14.120785 (+ 133us) client_negotiation.cc:240] Negotiation successful
0621 07:47:14.120975 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":116,"threads_started":1}
I20260621 07:47:14.122403 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 1958
I20260621 07:47:14.122521 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-0/wal/instance
I20260621 07:47:14.122752 771 external_mini_cluster.cc:1428] Setting key 314a82c8b1930f0760bbf9c21cb0c5ee
I20260621 07:47:14.122888 2089 heartbeater.cc:347] Connected to a master server at 127.0.192.254:44477
I20260621 07:47:14.123036 2089 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:14.123544 2089 heartbeater.cc:511] Master 127.0.192.254:44477 requested a full tablet report, sending...
I20260621 07:47:14.124971 1886 ts_manager.cc:194] Registered new tserver with Master: dc15feffe082457c99a95478676e263d (127.0.192.193:46215)
I20260621 07:47:14.126209 1886 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.193@KRBTEST.COM'} at 127.0.192.193:47799
I20260621 07:47:14.132370 1935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:14.125159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:46601 (local address 127.0.192.254:44477)
0621 07:47:14.125265 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:14.125270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:14.125282 (+ 12us) server_negotiation.cc:407] Connection header received
0621 07:47:14.125309 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:14.125311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:14.125349 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:14.125424 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:14.125875 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.126640 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.127916 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.128309 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.128392 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:14.129646 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:14.129648 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:14.129650 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:14.129669 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:14.130866 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:14.131285 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:14.131287 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:14.131288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:14.131349 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:14.131626 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:14.131628 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:14.131629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:14.131789 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:14.131878 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:14.132195 (+ 317us) server_negotiation.cc:299] Negotiation successful
0621 07:47:14.132275 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
WARNING: no policy specified for kudu/127.0.192.194@KRBTEST.COM; defaulting to no policy
I20260621 07:47:14.132472 2095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:14.124891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:44477 (local address 127.0.192.193:46601)
0621 07:47:14.125118 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:14.125131 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:14.125208 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:14.125438 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:14.125440 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:14.125605 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:14.125745 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:14.125753 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.126804 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.126810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:14.127786 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:14.127799 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.128548 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.128578 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:14.128678 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:14.129080 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:14.129099 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:14.129522 (+ 423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:14.130965 (+ 1443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:14.130968 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:14.130970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:14.131199 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:14.131433 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:14.131438 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:14.131441 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:14.131541 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:14.131915 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:14.131921 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:14.132030 (+ 109us) client_negotiation.cc:769] Sending connection context
0621 07:47:14.132142 (+ 112us) client_negotiation.cc:240] Negotiation successful
0621 07:47:14.132256 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":97,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab.
Entry for principal kudu/127.0.192.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab.
I20260621 07:47:14.147302 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.194:0
--local_ip_for_outbound_sockets=127.0.192.194
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:44477
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:39665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:14.246768 2098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:14.246950 2098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:14.247002 2098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:14.250443 2098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:14.250487 2098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:14.250519 2098 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 07:47:14.250552 2098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.194
I20260621 07:47:14.255496 2098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:39665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.192.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.192.254:44477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.194
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:14.256211 2098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:14.257019 2098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:14.265256 2103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:14.265256 2106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:14.265300 2104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:14.265642 2098 server_base.cc:1034] running on GCE node
I20260621 07:47:14.266048 2098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:14.266566 2098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:14.267732 2098 hybrid_clock.cc:648] HybridClock initialized: now 1782028034267712 us; error 33 us; skew 500 ppm
Jun 21 07:47:14 dist-test-slave-lzx3 krb5kdc[1841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:14.270133 2098 init.cc:377] Logged in from keytab as kudu/127.0.192.194@KRBTEST.COM (short username kudu)
I20260621 07:47:14.270869 2098 webserver.cc:460] Webserver started at http://127.0.192.194:45595/ using document root <none> and password file <none>
I20260621 07:47:14.271428 2098 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:14.271505 2098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:14.271705 2098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:14.273612 2098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-1/data/instance:
uuid: "1d6b8fc0a2c44be9a8a3394663274a3a"
format_stamp: "Formatted at 2026-06-21 07:47:14 on dist-test-slave-lzx3"
server_key: "a5066c5f2b1bebec19c0096c3f7cea5f"
server_key_iv: "0903dfe32d9b9520bb769b316812dceb"
server_key_version: "encryptionkey@0"
I20260621 07:47:14.274127 2098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-1/wal/instance:
uuid: "1d6b8fc0a2c44be9a8a3394663274a3a"
format_stamp: "Formatted at 2026-06-21 07:47:14 on dist-test-slave-lzx3"
server_key: "a5066c5f2b1bebec19c0096c3f7cea5f"
server_key_iv: "0903dfe32d9b9520bb769b316812dceb"
server_key_version: "encryptionkey@0"
I20260621 07:47:14.277350 2098 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:47:14.279345 2113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:14.280319 2098 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:14.280447 2098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-1/wal
uuid: "1d6b8fc0a2c44be9a8a3394663274a3a"
format_stamp: "Formatted at 2026-06-21 07:47:14 on dist-test-slave-lzx3"
server_key: "a5066c5f2b1bebec19c0096c3f7cea5f"
server_key_iv: "0903dfe32d9b9520bb769b316812dceb"
server_key_version: "encryptionkey@0"
I20260621 07:47:14.280539 2098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:14.299612 2098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:14.302541 2098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:14.302729 2098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:14.303966 2098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:47:14.304031 2098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:14.304085 2098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:47:14.304136 2098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 07:47:14 dist-test-slave-lzx3 krb5kdc[1841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:47:14.314165 2098 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.194:46767
I20260621 07:47:14.314237 2228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.194:46767 every 8 connection(s)
I20260621 07:47:14.315091 2098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
I20260621 07:47:14.317073 1935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:14.304602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:41581 (local address 127.0.192.254:44477)
0621 07:47:14.304719 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:14.304723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:14.306154 (+ 1431us) server_negotiation.cc:407] Connection header received
0621 07:47:14.307109 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:14.307115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:14.307180 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:14.307269 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:14.308524 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.309103 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.310192 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.310529 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.310657 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:14.313311 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:14.313316 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:14.313318 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:14.313349 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:14.314975 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:14.315523 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:14.315527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:14.315528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:14.315617 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:14.316007 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:14.316011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:14.316012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:14.316186 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:14.316291 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:14.316788 (+ 497us) server_negotiation.cc:299] Negotiation successful
0621 07:47:14.316903 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260621 07:47:14.317700 2122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:14.305490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:44477 (local address 127.0.192.194:41581)
0621 07:47:14.306008 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:14.306050 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:14.306877 (+ 827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:14.307419 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:14.307426 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:14.307858 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:14.308368 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:14.308380 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.309247 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.309250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:14.310064 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:14.310074 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.310694 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.310698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:14.310882 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:14.311454 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:14.311469 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:14.313113 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:14.315085 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:14.315089 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:14.315098 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:14.315392 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:14.315728 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:14.315732 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:14.315734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:14.315896 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:14.316300 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:14.316308 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:14.316511 (+ 203us) client_negotiation.cc:769] Sending connection context
0621 07:47:14.316741 (+ 230us) client_negotiation.cc:240] Negotiation successful
0621 07:47:14.317013 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":117,"threads_started":1}
I20260621 07:47:14.318753 2229 heartbeater.cc:347] Connected to a master server at 127.0.192.254:44477
I20260621 07:47:14.318894 2229 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:14.319281 2229 heartbeater.cc:511] Master 127.0.192.254:44477 requested a full tablet report, sending...
I20260621 07:47:14.320154 1885 ts_manager.cc:194] Registered new tserver with Master: 1d6b8fc0a2c44be9a8a3394663274a3a (127.0.192.194:46767)
I20260621 07:47:14.320819 1885 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.194@KRBTEST.COM'} at 127.0.192.194:41581
I20260621 07:47:14.321879 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 2098
I20260621 07:47:14.322002 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-1/wal/instance
I20260621 07:47:14.322317 771 external_mini_cluster.cc:1428] Setting key 8f2c46750131c1c633ea23461556c075
I20260621 07:47:14.327544 2235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:14.320573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:44477 (local address 127.0.192.194:40841)
0621 07:47:14.320837 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:14.320852 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:14.320935 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:14.321313 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:14.321316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:14.321465 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:14.321546 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:14.321552 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.322573 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.322577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:14.323162 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:14.323171 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.323716 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.323721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:14.323802 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:14.324185 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:14.324196 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:14.324436 (+ 240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:14.326155 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:14.326158 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:14.326160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:14.326342 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:14.326617 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:14.326618 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:14.326619 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:14.326726 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:14.327198 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:14.327200 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:14.327265 (+ 65us) client_negotiation.cc:769] Sending connection context
0621 07:47:14.327340 (+ 75us) client_negotiation.cc:240] Negotiation successful
0621 07:47:14.327429 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":105,"threads_started":1}
I20260621 07:47:14.327749 1935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:14.320624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:40841 (local address 127.0.192.254:44477)
0621 07:47:14.320743 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:14.320747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:14.320925 (+ 178us) server_negotiation.cc:407] Connection header received
0621 07:47:14.321165 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:14.321168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:14.321215 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:14.321276 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:14.321677 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.322434 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.323383 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.323596 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.323703 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:14.324579 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:14.324582 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:14.324584 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:14.324603 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:14.326039 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:14.326430 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:14.326436 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:14.326440 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:14.326521 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:14.326810 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:14.326816 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:14.326820 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:14.327104 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:14.327208 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:14.327427 (+ 219us) server_negotiation.cc:299] Negotiation successful
0621 07:47:14.327545 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
WARNING: no policy specified for kudu/127.0.192.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab.
Entry for principal kudu/127.0.192.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab.
I20260621 07:47:14.346319 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.195:0
--local_ip_for_outbound_sockets=127.0.192.195
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:44477
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:39665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:14.448901 2238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:14.449106 2238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:14.449143 2238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:14.452741 2238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:14.452790 2238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:14.452991 2238 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 07:47:14.453076 2238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.195
I20260621 07:47:14.457932 2238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:39665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.192.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.192.254:44477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.195
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:14.458731 2238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:14.459527 2238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:14.468178 2244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:14.468192 2243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:14.468192 2246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:14.468240 2238 server_base.cc:1034] running on GCE node
I20260621 07:47:14.468786 2238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:14.469255 2238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:14.470430 2238 hybrid_clock.cc:648] HybridClock initialized: now 1782028034470415 us; error 29 us; skew 500 ppm
Jun 21 07:47:14 dist-test-slave-lzx3 krb5kdc[1841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:14.473094 2238 init.cc:377] Logged in from keytab as kudu/127.0.192.195@KRBTEST.COM (short username kudu)
I20260621 07:47:14.473801 2238 webserver.cc:460] Webserver started at http://127.0.192.195:37867/ using document root <none> and password file <none>
I20260621 07:47:14.474294 2238 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:14.474377 2238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:14.474581 2238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:14.476343 2238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-2/data/instance:
uuid: "d0d9e88508924474886ee081ba8d6237"
format_stamp: "Formatted at 2026-06-21 07:47:14 on dist-test-slave-lzx3"
server_key: "28dec72282be67178fcdd2f1c389207b"
server_key_iv: "19d515cedf7f8f799b7818bc9a2aed46"
server_key_version: "encryptionkey@0"
I20260621 07:47:14.476879 2238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-2/wal/instance:
uuid: "d0d9e88508924474886ee081ba8d6237"
format_stamp: "Formatted at 2026-06-21 07:47:14 on dist-test-slave-lzx3"
server_key: "28dec72282be67178fcdd2f1c389207b"
server_key_iv: "19d515cedf7f8f799b7818bc9a2aed46"
server_key_version: "encryptionkey@0"
I20260621 07:47:14.479969 2238 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260621 07:47:14.481993 2253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:14.483083 2238 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 07:47:14.483210 2238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-2/wal
uuid: "d0d9e88508924474886ee081ba8d6237"
format_stamp: "Formatted at 2026-06-21 07:47:14 on dist-test-slave-lzx3"
server_key: "28dec72282be67178fcdd2f1c389207b"
server_key_iv: "19d515cedf7f8f799b7818bc9a2aed46"
server_key_version: "encryptionkey@0"
I20260621 07:47:14.483314 2238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:14.492789 2238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:14.495035 2238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:14.495204 2238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:14.496305 2238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:47:14.496347 2238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:14.496402 2238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:47:14.496430 2238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:14.505457 2238 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.195:37001
I20260621 07:47:14.505501 2368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.195:37001 every 8 connection(s)
Jun 21 07:47:14 dist-test-slave-lzx3 krb5kdc[1841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:47:14.506309 2238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
I20260621 07:47:14.509570 1935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:14.497095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:39343 (local address 127.0.192.254:44477)
0621 07:47:14.497205 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:14.497209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:14.498025 (+ 816us) server_negotiation.cc:407] Connection header received
0621 07:47:14.498922 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:14.498925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:14.498991 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:14.499093 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:14.500613 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.501509 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.503206 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.503592 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.503703 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:14.506271 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:14.506277 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:14.506283 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:14.506313 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:14.507697 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:14.508158 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:14.508163 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:14.508164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:14.508247 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:14.508614 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:14.508618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:14.508622 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:14.508781 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:14.508886 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:14.509299 (+ 413us) server_negotiation.cc:299] Negotiation successful
0621 07:47:14.509426 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260621 07:47:14.510154 2262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:14.497505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:44477 (local address 127.0.192.195:39343)
0621 07:47:14.497906 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:14.497938 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:14.498690 (+ 752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:14.499379 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:14.499387 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:14.499735 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:14.500385 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:14.500405 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.501634 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.501639 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:14.503090 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:14.503099 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.503750 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.503754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:14.503922 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:14.504597 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:14.504619 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:14.506111 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:14.507806 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:14.507811 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:14.507823 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:14.508057 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:14.508341 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:14.508344 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:14.508346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:14.508499 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:14.508885 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:14.508891 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:14.509069 (+ 178us) client_negotiation.cc:769] Sending connection context
0621 07:47:14.509253 (+ 184us) client_negotiation.cc:240] Negotiation successful
0621 07:47:14.509521 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":132,"threads_started":1}
I20260621 07:47:14.511147 2369 heartbeater.cc:347] Connected to a master server at 127.0.192.254:44477
I20260621 07:47:14.511291 2369 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:14.511410 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 2238
I20260621 07:47:14.511518 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-2/wal/instance
I20260621 07:47:14.511780 771 external_mini_cluster.cc:1428] Setting key 02f4ed08a8944d3da5e7f8dbe9a30a51
I20260621 07:47:14.511754 2369 heartbeater.cc:511] Master 127.0.192.254:44477 requested a full tablet report, sending...
I20260621 07:47:14.512914 1886 ts_manager.cc:194] Registered new tserver with Master: d0d9e88508924474886ee081ba8d6237 (127.0.192.195:37001)
I20260621 07:47:14.513489 1886 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.195@KRBTEST.COM'} at 127.0.192.195:39343
I20260621 07:47:14.513876 771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:47:14.521044 1935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:14.513947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:52211 (local address 127.0.192.254:44477)
0621 07:47:14.514052 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:14.514056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:14.514277 (+ 221us) server_negotiation.cc:407] Connection header received
0621 07:47:14.514372 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:14.514375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:14.514413 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:14.514483 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:14.514982 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.515530 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.516282 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.516617 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.516697 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:14.517955 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:14.517961 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:14.517966 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:14.517995 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:14.519553 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:14.519964 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:14.519968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:14.519969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:14.520050 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:14.520308 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:14.520311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:14.520312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:14.520457 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:14.520551 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:14.520817 (+ 266us) server_negotiation.cc:299] Negotiation successful
0621 07:47:14.520905 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260621 07:47:14.521041 2375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:14.513907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:44477 (local address 127.0.192.195:52211)
0621 07:47:14.514190 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:14.514202 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:14.514289 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:14.514515 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:14.514520 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:14.514714 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:14.514851 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:14.514860 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.515662 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.515667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:14.516178 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:14.516185 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.516708 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.516710 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:14.516767 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:14.517439 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:14.517451 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:14.517787 (+ 336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:14.519663 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:14.519668 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:14.519670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:14.519880 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:14.520139 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:14.520142 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:14.520144 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:14.520220 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:14.520586 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:14.520589 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:14.520666 (+ 77us) client_negotiation.cc:769] Sending connection context
0621 07:47:14.520769 (+ 103us) client_negotiation.cc:240] Negotiation successful
0621 07:47:14.520882 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":86,"threads_started":1}
Jun 21 07:47:14 dist-test-slave-lzx3 krb5kdc[1841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028034, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 07:47:14 dist-test-slave-lzx3 krb5kdc[1841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028034, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:47:14.546016 1935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:14.538129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57002 (local address 127.0.192.254:44477)
0621 07:47:14.538276 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:14.538280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:14.538294 (+ 14us) server_negotiation.cc:407] Connection header received
0621 07:47:14.538328 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:14.538330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:14.538386 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:14.538466 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:14.539409 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.539982 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.540845 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.541068 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.541186 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:14.543400 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:14.543404 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:14.543406 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:14.543428 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:14.544491 (+ 1063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:14.544895 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:14.544899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:14.544901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:14.544972 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:14.545222 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:14.545226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:14.545228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:14.545407 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:14.545531 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:14.545770 (+ 239us) server_negotiation.cc:299] Negotiation successful
0621 07:47:14.545870 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260621 07:47:14.548265 1886 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57002:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:47:14.551034 1886 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:47:14.563401 2386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:14.558229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.195:37001 (local address 127.0.0.1:41536)
0621 07:47:14.558511 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:14.558558 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:14.558661 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:14.559239 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:14.559244 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:14.559266 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:14.559363 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:14.559369 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.560519 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.560522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:14.561795 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:14.561803 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.562806 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.562809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:14.562913 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:14.562956 (+ 43us) client_negotiation.cc:769] Sending connection context
0621 07:47:14.563069 (+ 113us) client_negotiation.cc:240] Negotiation successful
0621 07:47:14.563204 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":31,"thread_start_us":78,"threads_started":1}
I20260621 07:47:14.563417 2388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:14.558817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:46767 (local address 127.0.0.1:42166)
0621 07:47:14.559041 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:14.559054 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:14.559126 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:14.559775 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:14.559778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:14.559794 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:14.559871 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:14.559876 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.561080 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.561083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:14.561809 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:14.561817 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.562861 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.562865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:14.563009 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:14.563067 (+ 58us) client_negotiation.cc:769] Sending connection context
0621 07:47:14.563147 (+ 80us) client_negotiation.cc:240] Negotiation successful
0621 07:47:14.563250 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":22,"thread_start_us":107,"threads_started":1}
I20260621 07:47:14.563611 2387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:14.558369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41536 (local address 127.0.192.195:37001)
0621 07:47:14.558681 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:14.558692 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:14.558712 (+ 20us) server_negotiation.cc:407] Connection header received
0621 07:47:14.558772 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:14.558776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:14.558946 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:14.559102 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:14.559482 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.560424 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.561900 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.562395 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.562521 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:14.562687 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:14.563340 (+ 653us) server_negotiation.cc:299] Negotiation successful
0621 07:47:14.563442 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":64,"threads_started":1}
I20260621 07:47:14.564220 2389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:14.558979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42166 (local address 127.0.192.194:46767)
0621 07:47:14.559366 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:14.559376 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:14.559396 (+ 20us) server_negotiation.cc:407] Connection header received
0621 07:47:14.559463 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:14.559469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:14.559634 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:14.559990 (+ 356us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:14.560121 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.560969 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.561935 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.562726 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.563536 (+ 810us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:14.563968 (+ 432us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:14.564012 (+ 44us) server_negotiation.cc:299] Negotiation successful
0621 07:47:14.564086 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":112,"threads_started":1}
I20260621 07:47:14.564527 2390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:14.559168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:46215 (local address 127.0.0.1:56422)
0621 07:47:14.559374 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:14.559387 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:14.559465 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:14.560228 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:14.560230 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:14.560246 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:14.560339 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:14.560347 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.561743 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.561746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:14.562877 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:14.562884 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.564192 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.564195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:14.564269 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:14.564287 (+ 18us) client_negotiation.cc:769] Sending connection context
0621 07:47:14.564338 (+ 51us) client_negotiation.cc:240] Negotiation successful
0621 07:47:14.564397 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":20,"thread_start_us":95,"threads_started":1}
I20260621 07:47:14.564695 2391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:14.559574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56422 (local address 127.0.192.193:46215)
0621 07:47:14.559834 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:14.559843 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:14.559862 (+ 19us) server_negotiation.cc:407] Connection header received
0621 07:47:14.559926 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:14.559930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:14.560092 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:14.560472 (+ 380us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:14.560630 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.561634 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.563120 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.564069 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.564174 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:14.564310 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:14.564380 (+ 70us) server_negotiation.cc:299] Negotiation successful
0621 07:47:14.564487 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":107,"threads_started":1}
I20260621 07:47:14.566241 2303 tablet_service.cc:1467] Processing CreateTablet for tablet 2d1620cad5074d54952db19347219dfb (DEFAULT_TABLE table=test-table [id=d17b6b9a14e44336917e08aa764d8929]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:47:14.567198 2303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d1620cad5074d54952db19347219dfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:14.566869 2163 tablet_service.cc:1467] Processing CreateTablet for tablet 2d1620cad5074d54952db19347219dfb (DEFAULT_TABLE table=test-table [id=d17b6b9a14e44336917e08aa764d8929]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:47:14.567134 2023 tablet_service.cc:1467] Processing CreateTablet for tablet 2d1620cad5074d54952db19347219dfb (DEFAULT_TABLE table=test-table [id=d17b6b9a14e44336917e08aa764d8929]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:47:14.567729 2163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d1620cad5074d54952db19347219dfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:14.567966 2023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d1620cad5074d54952db19347219dfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:14.574090 2392 tablet_bootstrap.cc:492] T 2d1620cad5074d54952db19347219dfb P dc15feffe082457c99a95478676e263d: Bootstrap starting.
I20260621 07:47:14.575345 2394 tablet_bootstrap.cc:492] T 2d1620cad5074d54952db19347219dfb P d0d9e88508924474886ee081ba8d6237: Bootstrap starting.
I20260621 07:47:14.576814 2392 tablet_bootstrap.cc:654] T 2d1620cad5074d54952db19347219dfb P dc15feffe082457c99a95478676e263d: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:14.576817 2393 tablet_bootstrap.cc:492] T 2d1620cad5074d54952db19347219dfb P 1d6b8fc0a2c44be9a8a3394663274a3a: Bootstrap starting.
I20260621 07:47:14.577222 2394 tablet_bootstrap.cc:654] T 2d1620cad5074d54952db19347219dfb P d0d9e88508924474886ee081ba8d6237: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:14.577595 2392 log.cc:824] T 2d1620cad5074d54952db19347219dfb P dc15feffe082457c99a95478676e263d: Log is configured to *not* fsync() on all Append() calls
I20260621 07:47:14.577960 2394 log.cc:824] T 2d1620cad5074d54952db19347219dfb P d0d9e88508924474886ee081ba8d6237: Log is configured to *not* fsync() on all Append() calls
I20260621 07:47:14.578481 2393 tablet_bootstrap.cc:654] T 2d1620cad5074d54952db19347219dfb P 1d6b8fc0a2c44be9a8a3394663274a3a: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:14.579037 2393 log.cc:824] T 2d1620cad5074d54952db19347219dfb P 1d6b8fc0a2c44be9a8a3394663274a3a: Log is configured to *not* fsync() on all Append() calls
I20260621 07:47:14.579228 2392 tablet_bootstrap.cc:492] T 2d1620cad5074d54952db19347219dfb P dc15feffe082457c99a95478676e263d: No bootstrap required, opened a new log
I20260621 07:47:14.579398 2392 ts_tablet_manager.cc:1397] T 2d1620cad5074d54952db19347219dfb P dc15feffe082457c99a95478676e263d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260621 07:47:14.580624 2393 tablet_bootstrap.cc:492] T 2d1620cad5074d54952db19347219dfb P 1d6b8fc0a2c44be9a8a3394663274a3a: No bootstrap required, opened a new log
I20260621 07:47:14.580785 2393 ts_tablet_manager.cc:1397] T 2d1620cad5074d54952db19347219dfb P 1d6b8fc0a2c44be9a8a3394663274a3a: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260621 07:47:14.582321 2392 raft_consensus.cc:348] T 2d1620cad5074d54952db19347219dfb P dc15feffe082457c99a95478676e263d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0d9e88508924474886ee081ba8d6237" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 37001 } } peers { permanent_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 46767 } } peers { permanent_uuid: "dc15feffe082457c99a95478676e263d" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46215 } }
I20260621 07:47:14.582538 2392 raft_consensus.cc:374] T 2d1620cad5074d54952db19347219dfb P dc15feffe082457c99a95478676e263d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:14.582594 2392 raft_consensus.cc:729] T 2d1620cad5074d54952db19347219dfb P dc15feffe082457c99a95478676e263d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc15feffe082457c99a95478676e263d, State: Initialized, Role: FOLLOWER
I20260621 07:47:14.582988 2392 consensus_queue.cc:260] T 2d1620cad5074d54952db19347219dfb P dc15feffe082457c99a95478676e263d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0d9e88508924474886ee081ba8d6237" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 37001 } } peers { permanent_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 46767 } } peers { permanent_uuid: "dc15feffe082457c99a95478676e263d" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46215 } }
I20260621 07:47:14.583431 2393 raft_consensus.cc:348] T 2d1620cad5074d54952db19347219dfb P 1d6b8fc0a2c44be9a8a3394663274a3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0d9e88508924474886ee081ba8d6237" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 37001 } } peers { permanent_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 46767 } } peers { permanent_uuid: "dc15feffe082457c99a95478676e263d" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46215 } }
I20260621 07:47:14.583616 2089 heartbeater.cc:503] Master 127.0.192.254:44477 was elected leader, sending a full tablet report...
I20260621 07:47:14.583678 2393 raft_consensus.cc:374] T 2d1620cad5074d54952db19347219dfb P 1d6b8fc0a2c44be9a8a3394663274a3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:14.583758 2393 raft_consensus.cc:729] T 2d1620cad5074d54952db19347219dfb P 1d6b8fc0a2c44be9a8a3394663274a3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d6b8fc0a2c44be9a8a3394663274a3a, State: Initialized, Role: FOLLOWER
I20260621 07:47:14.583920 2392 ts_tablet_manager.cc:1428] T 2d1620cad5074d54952db19347219dfb P dc15feffe082457c99a95478676e263d: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260621 07:47:14.584136 2393 consensus_queue.cc:260] T 2d1620cad5074d54952db19347219dfb P 1d6b8fc0a2c44be9a8a3394663274a3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0d9e88508924474886ee081ba8d6237" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 37001 } } peers { permanent_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 46767 } } peers { permanent_uuid: "dc15feffe082457c99a95478676e263d" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46215 } }
I20260621 07:47:14.584985 2393 ts_tablet_manager.cc:1428] T 2d1620cad5074d54952db19347219dfb P 1d6b8fc0a2c44be9a8a3394663274a3a: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260621 07:47:14.585100 2398 raft_consensus.cc:482] T 2d1620cad5074d54952db19347219dfb P dc15feffe082457c99a95478676e263d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:47:14.585281 2398 raft_consensus.cc:504] T 2d1620cad5074d54952db19347219dfb P dc15feffe082457c99a95478676e263d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0d9e88508924474886ee081ba8d6237" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 37001 } } peers { permanent_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 46767 } } peers { permanent_uuid: "dc15feffe082457c99a95478676e263d" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46215 } }
I20260621 07:47:14.585592 2394 tablet_bootstrap.cc:492] T 2d1620cad5074d54952db19347219dfb P d0d9e88508924474886ee081ba8d6237: No bootstrap required, opened a new log
I20260621 07:47:14.585742 2394 ts_tablet_manager.cc:1397] T 2d1620cad5074d54952db19347219dfb P d0d9e88508924474886ee081ba8d6237: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260621 07:47:14.586330 2229 heartbeater.cc:503] Master 127.0.192.254:44477 was elected leader, sending a full tablet report...
I20260621 07:47:14.589399 2394 raft_consensus.cc:348] T 2d1620cad5074d54952db19347219dfb P d0d9e88508924474886ee081ba8d6237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0d9e88508924474886ee081ba8d6237" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 37001 } } peers { permanent_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 46767 } } peers { permanent_uuid: "dc15feffe082457c99a95478676e263d" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46215 } }
I20260621 07:47:14.589655 2394 raft_consensus.cc:374] T 2d1620cad5074d54952db19347219dfb P d0d9e88508924474886ee081ba8d6237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:14.589723 2394 raft_consensus.cc:729] T 2d1620cad5074d54952db19347219dfb P d0d9e88508924474886ee081ba8d6237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0d9e88508924474886ee081ba8d6237, State: Initialized, Role: FOLLOWER
I20260621 07:47:14.590149 2394 consensus_queue.cc:260] T 2d1620cad5074d54952db19347219dfb P d0d9e88508924474886ee081ba8d6237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0d9e88508924474886ee081ba8d6237" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 37001 } } peers { permanent_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 46767 } } peers { permanent_uuid: "dc15feffe082457c99a95478676e263d" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46215 } }
I20260621 07:47:14.590777 2369 heartbeater.cc:503] Master 127.0.192.254:44477 was elected leader, sending a full tablet report...
I20260621 07:47:14.590899 1982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:14.586473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.195:37001 (local address 127.0.192.193:47297)
0621 07:47:14.586697 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:14.586710 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:14.586785 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:14.587334 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:14.587337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:14.587354 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:14.587455 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:14.587461 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.588423 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.588427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:14.589606 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:14.589618 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.590475 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.590479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:14.590598 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:14.590624 (+ 26us) client_negotiation.cc:769] Sending connection context
0621 07:47:14.590702 (+ 78us) client_negotiation.cc:240] Negotiation successful
0621 07:47:14.590772 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260621 07:47:14.591147 2394 ts_tablet_manager.cc:1428] T 2d1620cad5074d54952db19347219dfb P d0d9e88508924474886ee081ba8d6237: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260621 07:47:14.591125 2398 leader_election.cc:288] T 2d1620cad5074d54952db19347219dfb P dc15feffe082457c99a95478676e263d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0d9e88508924474886ee081ba8d6237 (127.0.192.195:37001), 1d6b8fc0a2c44be9a8a3394663274a3a (127.0.192.194:46767)
I20260621 07:47:14.591637 2387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:14.586916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:47297 (local address 127.0.192.195:37001)
0621 07:47:14.587062 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:14.587066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:14.587096 (+ 30us) server_negotiation.cc:407] Connection header received
0621 07:47:14.587134 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:14.587137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:14.587172 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:14.587242 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:14.587577 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.588293 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.589752 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.590338 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.590464 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:14.590509 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:14.591448 (+ 939us) server_negotiation.cc:299] Negotiation successful
0621 07:47:14.591516 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260621 07:47:14.592088 2323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d1620cad5074d54952db19347219dfb" candidate_uuid: "dc15feffe082457c99a95478676e263d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0d9e88508924474886ee081ba8d6237" is_pre_election: true
I20260621 07:47:14.592267 2323 raft_consensus.cc:2454] T 2d1620cad5074d54952db19347219dfb P d0d9e88508924474886ee081ba8d6237 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc15feffe082457c99a95478676e263d in term 0.
I20260621 07:47:14.592725 1977 leader_election.cc:302] T 2d1620cad5074d54952db19347219dfb P dc15feffe082457c99a95478676e263d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0d9e88508924474886ee081ba8d6237, dc15feffe082457c99a95478676e263d; no voters:
I20260621 07:47:14.592976 2398 raft_consensus.cc:2781] T 2d1620cad5074d54952db19347219dfb P dc15feffe082457c99a95478676e263d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:47:14.593140 2398 raft_consensus.cc:482] T 2d1620cad5074d54952db19347219dfb P dc15feffe082457c99a95478676e263d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:47:14.593256 2398 raft_consensus.cc:3037] T 2d1620cad5074d54952db19347219dfb P dc15feffe082457c99a95478676e263d [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:14.594430 2398 raft_consensus.cc:504] T 2d1620cad5074d54952db19347219dfb P dc15feffe082457c99a95478676e263d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0d9e88508924474886ee081ba8d6237" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 37001 } } peers { permanent_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 46767 } } peers { permanent_uuid: "dc15feffe082457c99a95478676e263d" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46215 } }
I20260621 07:47:14.594758 2398 leader_election.cc:288] T 2d1620cad5074d54952db19347219dfb P dc15feffe082457c99a95478676e263d [CANDIDATE]: Term 1 election: Requested vote from peers d0d9e88508924474886ee081ba8d6237 (127.0.192.195:37001), 1d6b8fc0a2c44be9a8a3394663274a3a (127.0.192.194:46767)
I20260621 07:47:14.595135 1982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:14.590830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:46767 (local address 127.0.192.193:33891)
0621 07:47:14.591033 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:14.591047 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:14.591112 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:14.591635 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:14.591638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:14.591654 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:14.591727 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:14.591732 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.592548 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.592552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:14.593557 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:14.593566 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.594399 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.594404 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:14.594494 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:14.594516 (+ 22us) client_negotiation.cc:769] Sending connection context
0621 07:47:14.594935 (+ 419us) client_negotiation.cc:240] Negotiation successful
0621 07:47:14.595018 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":85,"threads_started":1}
I20260621 07:47:14.595109 2323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d1620cad5074d54952db19347219dfb" candidate_uuid: "dc15feffe082457c99a95478676e263d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0d9e88508924474886ee081ba8d6237"
I20260621 07:47:14.595182 2389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:14.591323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:33891 (local address 127.0.192.194:46767)
0621 07:47:14.591427 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:14.591431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:14.591451 (+ 20us) server_negotiation.cc:407] Connection header received
0621 07:47:14.591484 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:14.591486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:14.591515 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:14.591636 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:14.591879 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.592391 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.593687 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.594271 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.594403 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:14.594456 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:14.594972 (+ 516us) server_negotiation.cc:299] Negotiation successful
0621 07:47:14.595064 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260621 07:47:14.595227 2323 raft_consensus.cc:3037] T 2d1620cad5074d54952db19347219dfb P d0d9e88508924474886ee081ba8d6237 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:14.595877 2183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d1620cad5074d54952db19347219dfb" candidate_uuid: "dc15feffe082457c99a95478676e263d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" is_pre_election: true
I20260621 07:47:14.595865 2182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d1620cad5074d54952db19347219dfb" candidate_uuid: "dc15feffe082457c99a95478676e263d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a"
I20260621 07:47:14.596081 2182 raft_consensus.cc:3037] T 2d1620cad5074d54952db19347219dfb P 1d6b8fc0a2c44be9a8a3394663274a3a [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:14.596335 2323 raft_consensus.cc:2454] T 2d1620cad5074d54952db19347219dfb P d0d9e88508924474886ee081ba8d6237 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc15feffe082457c99a95478676e263d in term 1.
I20260621 07:47:14.596868 1977 leader_election.cc:302] T 2d1620cad5074d54952db19347219dfb P dc15feffe082457c99a95478676e263d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0d9e88508924474886ee081ba8d6237, dc15feffe082457c99a95478676e263d; no voters:
I20260621 07:47:14.597131 2398 raft_consensus.cc:2781] T 2d1620cad5074d54952db19347219dfb P dc15feffe082457c99a95478676e263d [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:47:14.597365 2398 raft_consensus.cc:686] T 2d1620cad5074d54952db19347219dfb P dc15feffe082457c99a95478676e263d [term 1 LEADER]: Becoming Leader. State: Replica: dc15feffe082457c99a95478676e263d, State: Running, Role: LEADER
I20260621 07:47:14.597640 2398 consensus_queue.cc:237] T 2d1620cad5074d54952db19347219dfb P dc15feffe082457c99a95478676e263d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0d9e88508924474886ee081ba8d6237" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 37001 } } peers { permanent_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 46767 } } peers { permanent_uuid: "dc15feffe082457c99a95478676e263d" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46215 } }
I20260621 07:47:14.597700 2182 raft_consensus.cc:2454] T 2d1620cad5074d54952db19347219dfb P 1d6b8fc0a2c44be9a8a3394663274a3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc15feffe082457c99a95478676e263d in term 1.
I20260621 07:47:14.600317 1885 catalog_manager.cc:5495] T 2d1620cad5074d54952db19347219dfb P dc15feffe082457c99a95478676e263d reported cstate change: term changed from 0 to 1, leader changed from <none> to dc15feffe082457c99a95478676e263d (127.0.192.193). New cstate: current_term: 1 leader_uuid: "dc15feffe082457c99a95478676e263d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0d9e88508924474886ee081ba8d6237" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 37001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc15feffe082457c99a95478676e263d" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46215 } health_report { overall_health: HEALTHY } } }
I20260621 07:47:14.619329 2391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:14.616287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56424 (local address 127.0.192.193:46215)
0621 07:47:14.616416 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:14.616419 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:14.616431 (+ 12us) server_negotiation.cc:407] Connection header received
0621 07:47:14.616621 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:14.616623 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:14.616664 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:14.616737 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:47:14.616955 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.617386 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.618160 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.618338 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.618403 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:14.618648 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:47:14.618965 (+ 317us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:47:14.619041 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:14.619139 (+ 98us) server_negotiation.cc:299] Negotiation successful
0621 07:47:14.619187 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260621 07:47:14.619781 2090 tablet.cc:2369] T 2d1620cad5074d54952db19347219dfb P dc15feffe082457c99a95478676e263d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:47:14.624231 2182 raft_consensus.cc:1261] T 2d1620cad5074d54952db19347219dfb P 1d6b8fc0a2c44be9a8a3394663274a3a [term 1 FOLLOWER]: Refusing update from remote peer dc15feffe082457c99a95478676e263d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:47:14.624503 2323 raft_consensus.cc:1261] T 2d1620cad5074d54952db19347219dfb P d0d9e88508924474886ee081ba8d6237 [term 1 FOLLOWER]: Refusing update from remote peer dc15feffe082457c99a95478676e263d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:47:14.624998 2402 consensus_queue.cc:1035] T 2d1620cad5074d54952db19347219dfb P dc15feffe082457c99a95478676e263d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:47:14.625159 2398 consensus_queue.cc:1035] T 2d1620cad5074d54952db19347219dfb P dc15feffe082457c99a95478676e263d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0d9e88508924474886ee081ba8d6237" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 37001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:47:14.631220 2406 mvcc.cc:205] Tried to move back new op lower bound from 7299186829815291904 to 7299186829714792448. Current Snapshot: MvccSnapshot[applied={T|T < 7299186829815291904}]
I20260621 07:47:14.632082 2407 mvcc.cc:205] Tried to move back new op lower bound from 7299186829815291904 to 7299186829714792448. Current Snapshot: MvccSnapshot[applied={T|T < 7299186829815291904}]
I20260621 07:47:14.640399 1885 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:47:14.640547 1885 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:47:14.642750 1885 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c4822e5a59714dac88935d524bd2299a: Sending DeleteTablet for 3 replicas of tablet 2d1620cad5074d54952db19347219dfb
I20260621 07:47:14.643425 2303 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d1620cad5074d54952db19347219dfb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:47:14 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:41536
I20260621 07:47:14.643623 2163 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d1620cad5074d54952db19347219dfb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:47:14 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:42166
I20260621 07:47:14.643821 2023 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d1620cad5074d54952db19347219dfb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:47:14 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:56422
I20260621 07:47:14.643898 2414 tablet_replica.cc:330] T 2d1620cad5074d54952db19347219dfb P d0d9e88508924474886ee081ba8d6237: stopping tablet replica
I20260621 07:47:14.644107 2415 tablet_replica.cc:330] T 2d1620cad5074d54952db19347219dfb P 1d6b8fc0a2c44be9a8a3394663274a3a: stopping tablet replica
I20260621 07:47:14.644146 2414 raft_consensus.cc:2229] T 2d1620cad5074d54952db19347219dfb P d0d9e88508924474886ee081ba8d6237 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:47:14.644304 1885 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57002:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260621 07:47:14.644364 2415 raft_consensus.cc:2229] T 2d1620cad5074d54952db19347219dfb P 1d6b8fc0a2c44be9a8a3394663274a3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:47:14.644398 2414 raft_consensus.cc:2258] T 2d1620cad5074d54952db19347219dfb P d0d9e88508924474886ee081ba8d6237 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:47:14.644632 2415 raft_consensus.cc:2258] T 2d1620cad5074d54952db19347219dfb P 1d6b8fc0a2c44be9a8a3394663274a3a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260621 07:47:14.644692 1885 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:47:14.645193 2416 tablet_replica.cc:330] T 2d1620cad5074d54952db19347219dfb P dc15feffe082457c99a95478676e263d: stopping tablet replica
I20260621 07:47:14.645459 2416 raft_consensus.cc:2229] T 2d1620cad5074d54952db19347219dfb P dc15feffe082457c99a95478676e263d [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:47:14.645792 2416 raft_consensus.cc:2258] T 2d1620cad5074d54952db19347219dfb P dc15feffe082457c99a95478676e263d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:47:14.645988 2414 ts_tablet_manager.cc:1905] T 2d1620cad5074d54952db19347219dfb P d0d9e88508924474886ee081ba8d6237: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:47:14.646793 2415 ts_tablet_manager.cc:1905] T 2d1620cad5074d54952db19347219dfb P 1d6b8fc0a2c44be9a8a3394663274a3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:47:14.646879 2416 ts_tablet_manager.cc:1905] T 2d1620cad5074d54952db19347219dfb P dc15feffe082457c99a95478676e263d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:47:14.648994 2163 tablet_service.cc:1467] Processing CreateTablet for tablet b24a9656f2214c09aaed4ab5a8bdd32e (DEFAULT_TABLE table=test-table [id=435cbcffa30e40e69b56a496a8547432]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:47:14.649161 2415 ts_tablet_manager.cc:1918] T 2d1620cad5074d54952db19347219dfb P 1d6b8fc0a2c44be9a8a3394663274a3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:47:14.649173 2416 ts_tablet_manager.cc:1918] T 2d1620cad5074d54952db19347219dfb P dc15feffe082457c99a95478676e263d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:47:14.649209 2163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b24a9656f2214c09aaed4ab5a8bdd32e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:14.649226 2415 log.cc:1196] T 2d1620cad5074d54952db19347219dfb P 1d6b8fc0a2c44be9a8a3394663274a3a: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-1/wal/wals/2d1620cad5074d54952db19347219dfb
I20260621 07:47:14.649240 2416 log.cc:1196] T 2d1620cad5074d54952db19347219dfb P dc15feffe082457c99a95478676e263d: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-0/wal/wals/2d1620cad5074d54952db19347219dfb
I20260621 07:47:14.649498 2416 ts_tablet_manager.cc:1939] T 2d1620cad5074d54952db19347219dfb P dc15feffe082457c99a95478676e263d: Deleting consensus metadata
I20260621 07:47:14.649505 2415 ts_tablet_manager.cc:1939] T 2d1620cad5074d54952db19347219dfb P 1d6b8fc0a2c44be9a8a3394663274a3a: Deleting consensus metadata
I20260621 07:47:14.651153 2393 tablet_bootstrap.cc:492] T b24a9656f2214c09aaed4ab5a8bdd32e P 1d6b8fc0a2c44be9a8a3394663274a3a: Bootstrap starting.
I20260621 07:47:14.651650 1873 catalog_manager.cc:4841] TS 1d6b8fc0a2c44be9a8a3394663274a3a (127.0.192.194:46767): tablet 2d1620cad5074d54952db19347219dfb (table test-table [id=d17b6b9a14e44336917e08aa764d8929]) successfully deleted
I20260621 07:47:14.651803 1873 catalog_manager.cc:4841] TS dc15feffe082457c99a95478676e263d (127.0.192.193:46215): tablet 2d1620cad5074d54952db19347219dfb (table test-table [id=d17b6b9a14e44336917e08aa764d8929]) successfully deleted
I20260621 07:47:14.651973 2393 tablet_bootstrap.cc:654] T b24a9656f2214c09aaed4ab5a8bdd32e P 1d6b8fc0a2c44be9a8a3394663274a3a: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:14.652225 2303 tablet_service.cc:1467] Processing CreateTablet for tablet b24a9656f2214c09aaed4ab5a8bdd32e (DEFAULT_TABLE table=test-table [id=435cbcffa30e40e69b56a496a8547432]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:47:14.652437 2303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b24a9656f2214c09aaed4ab5a8bdd32e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:14.652727 2414 ts_tablet_manager.cc:1918] T 2d1620cad5074d54952db19347219dfb P d0d9e88508924474886ee081ba8d6237: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:47:14.652819 2414 log.cc:1196] T 2d1620cad5074d54952db19347219dfb P d0d9e88508924474886ee081ba8d6237: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-2/wal/wals/2d1620cad5074d54952db19347219dfb
I20260621 07:47:14.653110 2414 ts_tablet_manager.cc:1939] T 2d1620cad5074d54952db19347219dfb P d0d9e88508924474886ee081ba8d6237: Deleting consensus metadata
I20260621 07:47:14.654141 1873 catalog_manager.cc:4841] TS d0d9e88508924474886ee081ba8d6237 (127.0.192.195:37001): tablet 2d1620cad5074d54952db19347219dfb (table test-table [id=d17b6b9a14e44336917e08aa764d8929]) successfully deleted
I20260621 07:47:14.654247 2023 tablet_service.cc:1467] Processing CreateTablet for tablet b24a9656f2214c09aaed4ab5a8bdd32e (DEFAULT_TABLE table=test-table [id=435cbcffa30e40e69b56a496a8547432]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:47:14.654488 2023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b24a9656f2214c09aaed4ab5a8bdd32e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:14.654563 2394 tablet_bootstrap.cc:492] T b24a9656f2214c09aaed4ab5a8bdd32e P d0d9e88508924474886ee081ba8d6237: Bootstrap starting.
I20260621 07:47:14.655809 2394 tablet_bootstrap.cc:654] T b24a9656f2214c09aaed4ab5a8bdd32e P d0d9e88508924474886ee081ba8d6237: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:14.656002 2392 tablet_bootstrap.cc:492] T b24a9656f2214c09aaed4ab5a8bdd32e P dc15feffe082457c99a95478676e263d: Bootstrap starting.
I20260621 07:47:14.656901 2392 tablet_bootstrap.cc:654] T b24a9656f2214c09aaed4ab5a8bdd32e P dc15feffe082457c99a95478676e263d: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:14.657243 2394 tablet_bootstrap.cc:492] T b24a9656f2214c09aaed4ab5a8bdd32e P d0d9e88508924474886ee081ba8d6237: No bootstrap required, opened a new log
I20260621 07:47:14.657354 2394 ts_tablet_manager.cc:1397] T b24a9656f2214c09aaed4ab5a8bdd32e P d0d9e88508924474886ee081ba8d6237: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260621 07:47:14.657752 2394 raft_consensus.cc:348] T b24a9656f2214c09aaed4ab5a8bdd32e P d0d9e88508924474886ee081ba8d6237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc15feffe082457c99a95478676e263d" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46215 } } peers { permanent_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 46767 } } peers { permanent_uuid: "d0d9e88508924474886ee081ba8d6237" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 37001 } }
I20260621 07:47:14.657860 2394 raft_consensus.cc:374] T b24a9656f2214c09aaed4ab5a8bdd32e P d0d9e88508924474886ee081ba8d6237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:14.657913 2394 raft_consensus.cc:729] T b24a9656f2214c09aaed4ab5a8bdd32e P d0d9e88508924474886ee081ba8d6237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0d9e88508924474886ee081ba8d6237, State: Initialized, Role: FOLLOWER
I20260621 07:47:14.657964 2392 tablet_bootstrap.cc:492] T b24a9656f2214c09aaed4ab5a8bdd32e P dc15feffe082457c99a95478676e263d: No bootstrap required, opened a new log
I20260621 07:47:14.658033 2392 ts_tablet_manager.cc:1397] T b24a9656f2214c09aaed4ab5a8bdd32e P dc15feffe082457c99a95478676e263d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 07:47:14.658063 2394 consensus_queue.cc:260] T b24a9656f2214c09aaed4ab5a8bdd32e P d0d9e88508924474886ee081ba8d6237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc15feffe082457c99a95478676e263d" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46215 } } peers { permanent_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 46767 } } peers { permanent_uuid: "d0d9e88508924474886ee081ba8d6237" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 37001 } }
I20260621 07:47:14.658372 2394 ts_tablet_manager.cc:1428] T b24a9656f2214c09aaed4ab5a8bdd32e P d0d9e88508924474886ee081ba8d6237: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260621 07:47:14.658473 2392 raft_consensus.cc:348] T b24a9656f2214c09aaed4ab5a8bdd32e P dc15feffe082457c99a95478676e263d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc15feffe082457c99a95478676e263d" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46215 } } peers { permanent_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 46767 } } peers { permanent_uuid: "d0d9e88508924474886ee081ba8d6237" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 37001 } }
I20260621 07:47:14.658571 2392 raft_consensus.cc:374] T b24a9656f2214c09aaed4ab5a8bdd32e P dc15feffe082457c99a95478676e263d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:14.658603 2392 raft_consensus.cc:729] T b24a9656f2214c09aaed4ab5a8bdd32e P dc15feffe082457c99a95478676e263d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc15feffe082457c99a95478676e263d, State: Initialized, Role: FOLLOWER
I20260621 07:47:14.658702 2392 consensus_queue.cc:260] T b24a9656f2214c09aaed4ab5a8bdd32e P dc15feffe082457c99a95478676e263d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc15feffe082457c99a95478676e263d" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46215 } } peers { permanent_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 46767 } } peers { permanent_uuid: "d0d9e88508924474886ee081ba8d6237" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 37001 } }
I20260621 07:47:14.658916 2392 ts_tablet_manager.cc:1428] T b24a9656f2214c09aaed4ab5a8bdd32e P dc15feffe082457c99a95478676e263d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260621 07:47:14.659785 2393 tablet_bootstrap.cc:492] T b24a9656f2214c09aaed4ab5a8bdd32e P 1d6b8fc0a2c44be9a8a3394663274a3a: No bootstrap required, opened a new log
I20260621 07:47:14.659852 2393 ts_tablet_manager.cc:1397] T b24a9656f2214c09aaed4ab5a8bdd32e P 1d6b8fc0a2c44be9a8a3394663274a3a: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260621 07:47:14.660210 2393 raft_consensus.cc:348] T b24a9656f2214c09aaed4ab5a8bdd32e P 1d6b8fc0a2c44be9a8a3394663274a3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc15feffe082457c99a95478676e263d" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46215 } } peers { permanent_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 46767 } } peers { permanent_uuid: "d0d9e88508924474886ee081ba8d6237" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 37001 } }
I20260621 07:47:14.660290 2393 raft_consensus.cc:374] T b24a9656f2214c09aaed4ab5a8bdd32e P 1d6b8fc0a2c44be9a8a3394663274a3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:14.660316 2393 raft_consensus.cc:729] T b24a9656f2214c09aaed4ab5a8bdd32e P 1d6b8fc0a2c44be9a8a3394663274a3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d6b8fc0a2c44be9a8a3394663274a3a, State: Initialized, Role: FOLLOWER
I20260621 07:47:14.660409 2393 consensus_queue.cc:260] T b24a9656f2214c09aaed4ab5a8bdd32e P 1d6b8fc0a2c44be9a8a3394663274a3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc15feffe082457c99a95478676e263d" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46215 } } peers { permanent_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 46767 } } peers { permanent_uuid: "d0d9e88508924474886ee081ba8d6237" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 37001 } }
I20260621 07:47:14.660620 2393 ts_tablet_manager.cc:1428] T b24a9656f2214c09aaed4ab5a8bdd32e P 1d6b8fc0a2c44be9a8a3394663274a3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260621 07:47:14.757169 2370 tablet.cc:2369] T b24a9656f2214c09aaed4ab5a8bdd32e P d0d9e88508924474886ee081ba8d6237: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 07:47:14.816133 2230 tablet.cc:2369] T b24a9656f2214c09aaed4ab5a8bdd32e P 1d6b8fc0a2c44be9a8a3394663274a3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:47:14.856936 2400 raft_consensus.cc:482] T b24a9656f2214c09aaed4ab5a8bdd32e P d0d9e88508924474886ee081ba8d6237 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:47:14.857059 2400 raft_consensus.cc:504] T b24a9656f2214c09aaed4ab5a8bdd32e P d0d9e88508924474886ee081ba8d6237 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc15feffe082457c99a95478676e263d" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46215 } } peers { permanent_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 46767 } } peers { permanent_uuid: "d0d9e88508924474886ee081ba8d6237" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 37001 } }
I20260621 07:47:14.858101 2400 leader_election.cc:288] T b24a9656f2214c09aaed4ab5a8bdd32e P d0d9e88508924474886ee081ba8d6237 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc15feffe082457c99a95478676e263d (127.0.192.193:46215), 1d6b8fc0a2c44be9a8a3394663274a3a (127.0.192.194:46767)
I20260621 07:47:14.862648 2389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:14.858667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:37087 (local address 127.0.192.194:46767)
0621 07:47:14.858802 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:14.858807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:14.858974 (+ 167us) server_negotiation.cc:407] Connection header received
0621 07:47:14.859085 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:14.859088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:14.859135 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:14.859212 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:14.859433 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.860175 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.861427 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.861970 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.862092 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:14.862138 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:14.862447 (+ 309us) server_negotiation.cc:299] Negotiation successful
0621 07:47:14.862516 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 07:47:14.862655 2262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:14.858324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:46215 (local address 127.0.192.195:45307)
0621 07:47:14.858441 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:14.858457 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:14.858542 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:14.858852 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:14.858855 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:14.858873 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:14.859018 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:14.859026 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.860065 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.860067 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:14.861288 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:14.861299 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.862090 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.862094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:14.862268 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:14.862333 (+ 65us) client_negotiation.cc:769] Sending connection context
0621 07:47:14.862411 (+ 78us) client_negotiation.cc:240] Negotiation successful
0621 07:47:14.862537 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":139}
I20260621 07:47:14.862725 2391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:14.858386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:45307 (local address 127.0.192.193:46215)
0621 07:47:14.858517 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:14.858521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:14.858535 (+ 14us) server_negotiation.cc:407] Connection header received
0621 07:47:14.858664 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:14.858667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:14.858721 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:14.858801 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:14.859138 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.859968 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.861427 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.861970 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.862101 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:14.862138 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:14.862521 (+ 383us) server_negotiation.cc:299] Negotiation successful
0621 07:47:14.862598 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260621 07:47:14.862948 2420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:14.858604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:46767 (local address 127.0.192.195:37087)
0621 07:47:14.858866 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:14.858876 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:14.858986 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:14.859225 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:14.859227 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:14.859241 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:14.859327 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:14.859331 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.860254 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.860256 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:14.861318 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:14.861327 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.862100 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.862103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:14.862268 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:14.862332 (+ 64us) client_negotiation.cc:769] Sending connection context
0621 07:47:14.862411 (+ 79us) client_negotiation.cc:240] Negotiation successful
0621 07:47:14.862604 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":127,"thread_start_us":63,"threads_started":1}
I20260621 07:47:14.863179 2043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b24a9656f2214c09aaed4ab5a8bdd32e" candidate_uuid: "d0d9e88508924474886ee081ba8d6237" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc15feffe082457c99a95478676e263d" is_pre_election: true
I20260621 07:47:14.863447 2043 raft_consensus.cc:2454] T b24a9656f2214c09aaed4ab5a8bdd32e P dc15feffe082457c99a95478676e263d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0d9e88508924474886ee081ba8d6237 in term 0.
I20260621 07:47:14.863858 2182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b24a9656f2214c09aaed4ab5a8bdd32e" candidate_uuid: "d0d9e88508924474886ee081ba8d6237" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" is_pre_election: true
I20260621 07:47:14.863998 2182 raft_consensus.cc:2454] T b24a9656f2214c09aaed4ab5a8bdd32e P 1d6b8fc0a2c44be9a8a3394663274a3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0d9e88508924474886ee081ba8d6237 in term 0.
I20260621 07:47:14.864348 2257 leader_election.cc:302] T b24a9656f2214c09aaed4ab5a8bdd32e P d0d9e88508924474886ee081ba8d6237 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d6b8fc0a2c44be9a8a3394663274a3a, d0d9e88508924474886ee081ba8d6237; no voters:
I20260621 07:47:14.864643 2400 raft_consensus.cc:2781] T b24a9656f2214c09aaed4ab5a8bdd32e P d0d9e88508924474886ee081ba8d6237 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:47:14.864701 2400 raft_consensus.cc:482] T b24a9656f2214c09aaed4ab5a8bdd32e P d0d9e88508924474886ee081ba8d6237 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:47:14.864723 2400 raft_consensus.cc:3037] T b24a9656f2214c09aaed4ab5a8bdd32e P d0d9e88508924474886ee081ba8d6237 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:14.865674 2400 raft_consensus.cc:504] T b24a9656f2214c09aaed4ab5a8bdd32e P d0d9e88508924474886ee081ba8d6237 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc15feffe082457c99a95478676e263d" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46215 } } peers { permanent_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 46767 } } peers { permanent_uuid: "d0d9e88508924474886ee081ba8d6237" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 37001 } }
I20260621 07:47:14.865972 2400 leader_election.cc:288] T b24a9656f2214c09aaed4ab5a8bdd32e P d0d9e88508924474886ee081ba8d6237 [CANDIDATE]: Term 1 election: Requested vote from peers dc15feffe082457c99a95478676e263d (127.0.192.193:46215), 1d6b8fc0a2c44be9a8a3394663274a3a (127.0.192.194:46767)
I20260621 07:47:14.866338 2043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b24a9656f2214c09aaed4ab5a8bdd32e" candidate_uuid: "d0d9e88508924474886ee081ba8d6237" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc15feffe082457c99a95478676e263d"
I20260621 07:47:14.866402 2182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b24a9656f2214c09aaed4ab5a8bdd32e" candidate_uuid: "d0d9e88508924474886ee081ba8d6237" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a"
I20260621 07:47:14.866441 2043 raft_consensus.cc:3037] T b24a9656f2214c09aaed4ab5a8bdd32e P dc15feffe082457c99a95478676e263d [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:14.866497 2182 raft_consensus.cc:3037] T b24a9656f2214c09aaed4ab5a8bdd32e P 1d6b8fc0a2c44be9a8a3394663274a3a [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:14.867452 2182 raft_consensus.cc:2454] T b24a9656f2214c09aaed4ab5a8bdd32e P 1d6b8fc0a2c44be9a8a3394663274a3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0d9e88508924474886ee081ba8d6237 in term 1.
I20260621 07:47:14.867452 2043 raft_consensus.cc:2454] T b24a9656f2214c09aaed4ab5a8bdd32e P dc15feffe082457c99a95478676e263d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0d9e88508924474886ee081ba8d6237 in term 1.
I20260621 07:47:14.867688 2257 leader_election.cc:302] T b24a9656f2214c09aaed4ab5a8bdd32e P d0d9e88508924474886ee081ba8d6237 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0d9e88508924474886ee081ba8d6237, dc15feffe082457c99a95478676e263d; no voters:
I20260621 07:47:14.867844 2400 raft_consensus.cc:2781] T b24a9656f2214c09aaed4ab5a8bdd32e P d0d9e88508924474886ee081ba8d6237 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:47:14.868019 2400 raft_consensus.cc:686] T b24a9656f2214c09aaed4ab5a8bdd32e P d0d9e88508924474886ee081ba8d6237 [term 1 LEADER]: Becoming Leader. State: Replica: d0d9e88508924474886ee081ba8d6237, State: Running, Role: LEADER
I20260621 07:47:14.868237 2400 consensus_queue.cc:237] T b24a9656f2214c09aaed4ab5a8bdd32e P d0d9e88508924474886ee081ba8d6237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc15feffe082457c99a95478676e263d" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46215 } } peers { permanent_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 46767 } } peers { permanent_uuid: "d0d9e88508924474886ee081ba8d6237" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 37001 } }
I20260621 07:47:14.870034 1884 catalog_manager.cc:5495] T b24a9656f2214c09aaed4ab5a8bdd32e P d0d9e88508924474886ee081ba8d6237 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0d9e88508924474886ee081ba8d6237 (127.0.192.195). New cstate: current_term: 1 leader_uuid: "d0d9e88508924474886ee081ba8d6237" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc15feffe082457c99a95478676e263d" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0d9e88508924474886ee081ba8d6237" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 37001 } health_report { overall_health: HEALTHY } } }
Jun 21 07:47:14 dist-test-slave-lzx3 krb5kdc[1841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:47:14.913193 2428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:14.905792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:44477 (local address 127.0.0.1:57004)
0621 07:47:14.906065 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:14.906076 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:14.906182 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:14.906384 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:14.906386 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:14.906728 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:14.906798 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:14.906803 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.907461 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.907466 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:14.907931 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:14.907937 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.908319 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.908322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:14.908400 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:14.908897 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:14.908908 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:14.910089 (+ 1181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:14.911525 (+ 1436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:14.911533 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:14.911538 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:14.911747 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:14.912113 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:14.912116 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:14.912119 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:14.912235 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:14.912624 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:14.912631 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:14.912734 (+ 103us) client_negotiation.cc:769] Sending connection context
0621 07:47:14.912851 (+ 117us) client_negotiation.cc:240] Negotiation successful
0621 07:47:14.912990 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":2,"thread_start_us":84,"threads_started":1}
I20260621 07:47:14.913193 1935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:14.905891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57004 (local address 127.0.192.254:44477)
0621 07:47:14.906016 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:14.906020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:14.906147 (+ 127us) server_negotiation.cc:407] Connection header received
0621 07:47:14.906253 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:14.906256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:14.906298 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:14.906368 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:14.906907 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.907368 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.908034 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.908225 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.908317 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:14.910210 (+ 1893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:14.910213 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:14.910215 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:14.910240 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:14.911403 (+ 1163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:14.911940 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:14.911944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:14.911945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:14.912022 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:14.912321 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:14.912327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:14.912328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:14.912487 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:14.912624 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:14.912896 (+ 272us) server_negotiation.cc:299] Negotiation successful
0621 07:47:14.913044 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":57}
I20260621 07:47:14.916461 1884 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:57004:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260621 07:47:14.917088 1884 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:47:14.921016 2023 tablet_service.cc:1467] Processing CreateTablet for tablet f411bbc269104667ae649f34867b9238 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6ad1442d63f34a24b85f019b68e01413]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 07:47:14.921130 2163 tablet_service.cc:1467] Processing CreateTablet for tablet f411bbc269104667ae649f34867b9238 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6ad1442d63f34a24b85f019b68e01413]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 07:47:14.921116 2303 tablet_service.cc:1467] Processing CreateTablet for tablet f411bbc269104667ae649f34867b9238 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6ad1442d63f34a24b85f019b68e01413]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 07:47:14.921391 2023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f411bbc269104667ae649f34867b9238. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:14.921440 2163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f411bbc269104667ae649f34867b9238. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:14.921440 2303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f411bbc269104667ae649f34867b9238. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:14.923552 2392 tablet_bootstrap.cc:492] T f411bbc269104667ae649f34867b9238 P dc15feffe082457c99a95478676e263d: Bootstrap starting.
I20260621 07:47:14.924057 2394 tablet_bootstrap.cc:492] T f411bbc269104667ae649f34867b9238 P d0d9e88508924474886ee081ba8d6237: Bootstrap starting.
I20260621 07:47:14.924335 2392 tablet_bootstrap.cc:654] T f411bbc269104667ae649f34867b9238 P dc15feffe082457c99a95478676e263d: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:14.924834 2394 tablet_bootstrap.cc:654] T f411bbc269104667ae649f34867b9238 P d0d9e88508924474886ee081ba8d6237: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:14.925238 2392 tablet_bootstrap.cc:492] T f411bbc269104667ae649f34867b9238 P dc15feffe082457c99a95478676e263d: No bootstrap required, opened a new log
I20260621 07:47:14.925334 2392 ts_tablet_manager.cc:1397] T f411bbc269104667ae649f34867b9238 P dc15feffe082457c99a95478676e263d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 07:47:14.925666 2392 raft_consensus.cc:348] T f411bbc269104667ae649f34867b9238 P dc15feffe082457c99a95478676e263d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc15feffe082457c99a95478676e263d" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46215 } } peers { permanent_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 46767 } } peers { permanent_uuid: "d0d9e88508924474886ee081ba8d6237" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 37001 } }
I20260621 07:47:14.925766 2392 raft_consensus.cc:374] T f411bbc269104667ae649f34867b9238 P dc15feffe082457c99a95478676e263d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:14.925803 2392 raft_consensus.cc:729] T f411bbc269104667ae649f34867b9238 P dc15feffe082457c99a95478676e263d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc15feffe082457c99a95478676e263d, State: Initialized, Role: FOLLOWER
I20260621 07:47:14.925958 2392 consensus_queue.cc:260] T f411bbc269104667ae649f34867b9238 P dc15feffe082457c99a95478676e263d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc15feffe082457c99a95478676e263d" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46215 } } peers { permanent_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 46767 } } peers { permanent_uuid: "d0d9e88508924474886ee081ba8d6237" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 37001 } }
I20260621 07:47:14.926198 2392 ts_tablet_manager.cc:1428] T f411bbc269104667ae649f34867b9238 P dc15feffe082457c99a95478676e263d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260621 07:47:14.926290 2394 tablet_bootstrap.cc:492] T f411bbc269104667ae649f34867b9238 P d0d9e88508924474886ee081ba8d6237: No bootstrap required, opened a new log
I20260621 07:47:14.926373 2394 ts_tablet_manager.cc:1397] T f411bbc269104667ae649f34867b9238 P d0d9e88508924474886ee081ba8d6237: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:14.926378 2402 tablet_replica.cc:439] T f411bbc269104667ae649f34867b9238 P dc15feffe082457c99a95478676e263d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc15feffe082457c99a95478676e263d" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46215 } } peers { permanent_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 46767 } } peers { permanent_uuid: "d0d9e88508924474886ee081ba8d6237" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 37001 } } }
I20260621 07:47:14.926487 2402 tablet_replica.cc:442] T f411bbc269104667ae649f34867b9238 P dc15feffe082457c99a95478676e263d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:47:14.926867 2394 raft_consensus.cc:348] T f411bbc269104667ae649f34867b9238 P d0d9e88508924474886ee081ba8d6237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc15feffe082457c99a95478676e263d" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46215 } } peers { permanent_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 46767 } } peers { permanent_uuid: "d0d9e88508924474886ee081ba8d6237" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 37001 } }
I20260621 07:47:14.926985 2394 raft_consensus.cc:374] T f411bbc269104667ae649f34867b9238 P d0d9e88508924474886ee081ba8d6237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:14.927019 2394 raft_consensus.cc:729] T f411bbc269104667ae649f34867b9238 P d0d9e88508924474886ee081ba8d6237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0d9e88508924474886ee081ba8d6237, State: Initialized, Role: FOLLOWER
I20260621 07:47:14.927142 2394 consensus_queue.cc:260] T f411bbc269104667ae649f34867b9238 P d0d9e88508924474886ee081ba8d6237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc15feffe082457c99a95478676e263d" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46215 } } peers { permanent_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 46767 } } peers { permanent_uuid: "d0d9e88508924474886ee081ba8d6237" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 37001 } }
I20260621 07:47:14.927402 2400 tablet_replica.cc:439] T f411bbc269104667ae649f34867b9238 P d0d9e88508924474886ee081ba8d6237: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc15feffe082457c99a95478676e263d" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46215 } } peers { permanent_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 46767 } } peers { permanent_uuid: "d0d9e88508924474886ee081ba8d6237" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 37001 } } }
I20260621 07:47:14.927523 2400 tablet_replica.cc:442] T f411bbc269104667ae649f34867b9238 P d0d9e88508924474886ee081ba8d6237: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:47:14.927635 2393 tablet_bootstrap.cc:492] T f411bbc269104667ae649f34867b9238 P 1d6b8fc0a2c44be9a8a3394663274a3a: Bootstrap starting.
I20260621 07:47:14.927459 2394 ts_tablet_manager.cc:1428] T f411bbc269104667ae649f34867b9238 P d0d9e88508924474886ee081ba8d6237: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260621 07:47:14.928529 2393 tablet_bootstrap.cc:654] T f411bbc269104667ae649f34867b9238 P 1d6b8fc0a2c44be9a8a3394663274a3a: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:14.929704 2393 tablet_bootstrap.cc:492] T f411bbc269104667ae649f34867b9238 P 1d6b8fc0a2c44be9a8a3394663274a3a: No bootstrap required, opened a new log
I20260621 07:47:14.929764 2393 ts_tablet_manager.cc:1397] T f411bbc269104667ae649f34867b9238 P 1d6b8fc0a2c44be9a8a3394663274a3a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260621 07:47:14.930191 2393 raft_consensus.cc:348] T f411bbc269104667ae649f34867b9238 P 1d6b8fc0a2c44be9a8a3394663274a3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc15feffe082457c99a95478676e263d" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46215 } } peers { permanent_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 46767 } } peers { permanent_uuid: "d0d9e88508924474886ee081ba8d6237" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 37001 } }
I20260621 07:47:14.930294 2393 raft_consensus.cc:374] T f411bbc269104667ae649f34867b9238 P 1d6b8fc0a2c44be9a8a3394663274a3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:14.930353 2393 raft_consensus.cc:729] T f411bbc269104667ae649f34867b9238 P 1d6b8fc0a2c44be9a8a3394663274a3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d6b8fc0a2c44be9a8a3394663274a3a, State: Initialized, Role: FOLLOWER
I20260621 07:47:14.930477 2393 consensus_queue.cc:260] T f411bbc269104667ae649f34867b9238 P 1d6b8fc0a2c44be9a8a3394663274a3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc15feffe082457c99a95478676e263d" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46215 } } peers { permanent_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 46767 } } peers { permanent_uuid: "d0d9e88508924474886ee081ba8d6237" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 37001 } }
I20260621 07:47:14.930716 2393 ts_tablet_manager.cc:1428] T f411bbc269104667ae649f34867b9238 P 1d6b8fc0a2c44be9a8a3394663274a3a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260621 07:47:14.930706 2399 tablet_replica.cc:439] T f411bbc269104667ae649f34867b9238 P 1d6b8fc0a2c44be9a8a3394663274a3a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc15feffe082457c99a95478676e263d" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46215 } } peers { permanent_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 46767 } } peers { permanent_uuid: "d0d9e88508924474886ee081ba8d6237" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 37001 } } }
I20260621 07:47:14.930881 2399 tablet_replica.cc:442] T f411bbc269104667ae649f34867b9238 P 1d6b8fc0a2c44be9a8a3394663274a3a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:47:14.943786 2399 raft_consensus.cc:482] T f411bbc269104667ae649f34867b9238 P 1d6b8fc0a2c44be9a8a3394663274a3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:47:14.943998 2399 raft_consensus.cc:504] T f411bbc269104667ae649f34867b9238 P 1d6b8fc0a2c44be9a8a3394663274a3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc15feffe082457c99a95478676e263d" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46215 } } peers { permanent_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 46767 } } peers { permanent_uuid: "d0d9e88508924474886ee081ba8d6237" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 37001 } }
I20260621 07:47:14.944998 2399 leader_election.cc:288] T f411bbc269104667ae649f34867b9238 P 1d6b8fc0a2c44be9a8a3394663274a3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc15feffe082457c99a95478676e263d (127.0.192.193:46215), d0d9e88508924474886ee081ba8d6237 (127.0.192.195:37001)
I20260621 07:47:14.948758 2391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:14.945232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:51107 (local address 127.0.192.193:46215)
0621 07:47:14.945350 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:14.945357 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:14.945484 (+ 127us) server_negotiation.cc:407] Connection header received
0621 07:47:14.945728 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:14.945732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:14.945786 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:14.945879 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:14.946156 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.946650 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.947741 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.948132 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.948266 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:14.948317 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:14.948529 (+ 212us) server_negotiation.cc:299] Negotiation successful
0621 07:47:14.948649 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260621 07:47:14.948838 2433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:14.945536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.195:37001 (local address 127.0.192.194:47357)
0621 07:47:14.945802 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:14.945813 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:14.945924 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:14.946170 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:14.946173 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:14.946187 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:14.946299 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:14.946303 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.946971 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.946975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:14.947766 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:14.947774 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.948370 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.948374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:14.948523 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:14.948546 (+ 23us) client_negotiation.cc:769] Sending connection context
0621 07:47:14.948639 (+ 93us) client_negotiation.cc:240] Negotiation successful
0621 07:47:14.948701 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":37,"thread_start_us":75,"threads_started":1}
I20260621 07:47:14.948858 2387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:14.945630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:47357 (local address 127.0.192.195:37001)
0621 07:47:14.945743 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:14.945747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:14.945887 (+ 140us) server_negotiation.cc:407] Connection header received
0621 07:47:14.946039 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:14.946042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:14.946079 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:14.946163 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:14.946398 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.946864 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.947880 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.948256 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.948375 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:14.948427 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:14.948659 (+ 232us) server_negotiation.cc:299] Negotiation successful
0621 07:47:14.948735 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260621 07:47:14.949254 2323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f411bbc269104667ae649f34867b9238" candidate_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0d9e88508924474886ee081ba8d6237" is_pre_election: true
I20260621 07:47:14.949332 2432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:14.945164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:46215 (local address 127.0.192.194:51107)
0621 07:47:14.945382 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:14.945395 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:14.945527 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:14.945907 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:14.945910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:14.945928 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:14.946047 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:14.946052 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.946767 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.946774 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:14.947605 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:14.947615 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.948256 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.948260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:14.948358 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:14.948388 (+ 30us) client_negotiation.cc:769] Sending connection context
0621 07:47:14.948516 (+ 128us) client_negotiation.cc:240] Negotiation successful
0621 07:47:14.949222 (+ 706us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":2,"thread_start_us":89,"threads_started":1}
I20260621 07:47:14.949395 2323 raft_consensus.cc:2454] T f411bbc269104667ae649f34867b9238 P d0d9e88508924474886ee081ba8d6237 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d6b8fc0a2c44be9a8a3394663274a3a in term 0.
I20260621 07:47:14.949690 2117 leader_election.cc:302] T f411bbc269104667ae649f34867b9238 P 1d6b8fc0a2c44be9a8a3394663274a3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d6b8fc0a2c44be9a8a3394663274a3a, d0d9e88508924474886ee081ba8d6237; no voters:
I20260621 07:47:14.949713 2043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f411bbc269104667ae649f34867b9238" candidate_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc15feffe082457c99a95478676e263d" is_pre_election: true
I20260621 07:47:14.949836 2043 raft_consensus.cc:2454] T f411bbc269104667ae649f34867b9238 P dc15feffe082457c99a95478676e263d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d6b8fc0a2c44be9a8a3394663274a3a in term 0.
I20260621 07:47:14.949925 2399 raft_consensus.cc:2781] T f411bbc269104667ae649f34867b9238 P 1d6b8fc0a2c44be9a8a3394663274a3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:47:14.949990 2399 raft_consensus.cc:482] T f411bbc269104667ae649f34867b9238 P 1d6b8fc0a2c44be9a8a3394663274a3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:47:14.950039 2399 raft_consensus.cc:3037] T f411bbc269104667ae649f34867b9238 P 1d6b8fc0a2c44be9a8a3394663274a3a [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:14.951387 2399 raft_consensus.cc:504] T f411bbc269104667ae649f34867b9238 P 1d6b8fc0a2c44be9a8a3394663274a3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc15feffe082457c99a95478676e263d" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46215 } } peers { permanent_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 46767 } } peers { permanent_uuid: "d0d9e88508924474886ee081ba8d6237" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 37001 } }
I20260621 07:47:14.951709 2399 leader_election.cc:288] T f411bbc269104667ae649f34867b9238 P 1d6b8fc0a2c44be9a8a3394663274a3a [CANDIDATE]: Term 1 election: Requested vote from peers dc15feffe082457c99a95478676e263d (127.0.192.193:46215), d0d9e88508924474886ee081ba8d6237 (127.0.192.195:37001)
I20260621 07:47:14.952060 2043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f411bbc269104667ae649f34867b9238" candidate_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc15feffe082457c99a95478676e263d"
I20260621 07:47:14.952165 2043 raft_consensus.cc:3037] T f411bbc269104667ae649f34867b9238 P dc15feffe082457c99a95478676e263d [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:14.952126 2323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f411bbc269104667ae649f34867b9238" candidate_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0d9e88508924474886ee081ba8d6237"
I20260621 07:47:14.952265 2323 raft_consensus.cc:3037] T f411bbc269104667ae649f34867b9238 P d0d9e88508924474886ee081ba8d6237 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:14.953059 2043 raft_consensus.cc:2454] T f411bbc269104667ae649f34867b9238 P dc15feffe082457c99a95478676e263d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d6b8fc0a2c44be9a8a3394663274a3a in term 1.
I20260621 07:47:14.953198 2323 raft_consensus.cc:2454] T f411bbc269104667ae649f34867b9238 P d0d9e88508924474886ee081ba8d6237 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d6b8fc0a2c44be9a8a3394663274a3a in term 1.
I20260621 07:47:14.953336 2117 leader_election.cc:302] T f411bbc269104667ae649f34867b9238 P 1d6b8fc0a2c44be9a8a3394663274a3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d6b8fc0a2c44be9a8a3394663274a3a, dc15feffe082457c99a95478676e263d; no voters:
I20260621 07:47:14.953512 2399 raft_consensus.cc:2781] T f411bbc269104667ae649f34867b9238 P 1d6b8fc0a2c44be9a8a3394663274a3a [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:47:14.953701 2399 raft_consensus.cc:686] T f411bbc269104667ae649f34867b9238 P 1d6b8fc0a2c44be9a8a3394663274a3a [term 1 LEADER]: Becoming Leader. State: Replica: 1d6b8fc0a2c44be9a8a3394663274a3a, State: Running, Role: LEADER
I20260621 07:47:14.953928 2399 consensus_queue.cc:237] T f411bbc269104667ae649f34867b9238 P 1d6b8fc0a2c44be9a8a3394663274a3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc15feffe082457c99a95478676e263d" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46215 } } peers { permanent_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 46767 } } peers { permanent_uuid: "d0d9e88508924474886ee081ba8d6237" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 37001 } }
I20260621 07:47:14.954994 2434 tablet_replica.cc:439] T f411bbc269104667ae649f34867b9238 P 1d6b8fc0a2c44be9a8a3394663274a3a: TxnStatusTablet state changed. Reason: New leader 1d6b8fc0a2c44be9a8a3394663274a3a. Latest consensus state: current_term: 1 leader_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc15feffe082457c99a95478676e263d" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46215 } } peers { permanent_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 46767 } } peers { permanent_uuid: "d0d9e88508924474886ee081ba8d6237" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 37001 } } }
I20260621 07:47:14.955101 2434 tablet_replica.cc:442] T f411bbc269104667ae649f34867b9238 P 1d6b8fc0a2c44be9a8a3394663274a3a: This TxnStatusTablet replica's current role is: LEADER
I20260621 07:47:14.955621 2436 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260621 07:47:14.955765 2436 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260621 07:47:14.956014 1885 catalog_manager.cc:5495] T f411bbc269104667ae649f34867b9238 P 1d6b8fc0a2c44be9a8a3394663274a3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d6b8fc0a2c44be9a8a3394663274a3a (127.0.192.194). New cstate: current_term: 1 leader_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc15feffe082457c99a95478676e263d" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0d9e88508924474886ee081ba8d6237" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 37001 } health_report { overall_health: UNKNOWN } } }
I20260621 07:47:14.973682 2389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:14.969772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42180 (local address 127.0.192.194:46767)
0621 07:47:14.969885 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:14.969888 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:14.969978 (+ 90us) server_negotiation.cc:407] Connection header received
0621 07:47:14.970143 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:14.970145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:14.970193 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:14.970278 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:47:14.970526 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.970992 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.972263 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.972475 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.972629 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:14.972899 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:47:14.973331 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:47:14.973423 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:14.973532 (+ 109us) server_negotiation.cc:299] Negotiation successful
0621 07:47:14.973591 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260621 07:47:14.973734 2428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:14.969694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:46767 (local address 127.0.0.1:42180)
0621 07:47:14.969904 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:14.969918 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:14.970021 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:14.970308 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:14.970311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:14.970318 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 07:47:14.970413 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:14.970419 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.971088 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.971092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:14.972098 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:14.972109 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.972612 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.972616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:14.972712 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:14.972769 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 07:47:14.973429 (+ 660us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 07:47:14.973444 (+ 15us) client_negotiation.cc:769] Sending connection context
0621 07:47:14.973523 (+ 79us) client_negotiation.cc:240] Negotiation successful
0621 07:47:14.973601 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125}
I20260621 07:47:14.976313 2323 raft_consensus.cc:1261] T f411bbc269104667ae649f34867b9238 P d0d9e88508924474886ee081ba8d6237 [term 1 FOLLOWER]: Refusing update from remote peer 1d6b8fc0a2c44be9a8a3394663274a3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:47:14.976431 2043 raft_consensus.cc:1261] T f411bbc269104667ae649f34867b9238 P dc15feffe082457c99a95478676e263d [term 1 FOLLOWER]: Refusing update from remote peer 1d6b8fc0a2c44be9a8a3394663274a3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:47:14.976817 2399 consensus_queue.cc:1035] T f411bbc269104667ae649f34867b9238 P 1d6b8fc0a2c44be9a8a3394663274a3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0d9e88508924474886ee081ba8d6237" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 37001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:47:14.976996 2434 consensus_queue.cc:1035] T f411bbc269104667ae649f34867b9238 P 1d6b8fc0a2c44be9a8a3394663274a3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc15feffe082457c99a95478676e263d" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:47:14.978492 2402 tablet_replica.cc:439] T f411bbc269104667ae649f34867b9238 P dc15feffe082457c99a95478676e263d: TxnStatusTablet state changed. Reason: New leader 1d6b8fc0a2c44be9a8a3394663274a3a. Latest consensus state: current_term: 1 leader_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc15feffe082457c99a95478676e263d" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46215 } } peers { permanent_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 46767 } } peers { permanent_uuid: "d0d9e88508924474886ee081ba8d6237" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 37001 } } }
I20260621 07:47:14.978602 2402 tablet_replica.cc:442] T f411bbc269104667ae649f34867b9238 P dc15feffe082457c99a95478676e263d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:47:14.979540 2400 tablet_replica.cc:439] T f411bbc269104667ae649f34867b9238 P d0d9e88508924474886ee081ba8d6237: TxnStatusTablet state changed. Reason: New leader 1d6b8fc0a2c44be9a8a3394663274a3a. Latest consensus state: current_term: 1 leader_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc15feffe082457c99a95478676e263d" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46215 } } peers { permanent_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 46767 } } peers { permanent_uuid: "d0d9e88508924474886ee081ba8d6237" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 37001 } } }
I20260621 07:47:14.979655 2400 tablet_replica.cc:442] T f411bbc269104667ae649f34867b9238 P d0d9e88508924474886ee081ba8d6237: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:47:14.980532 2434 tablet_replica.cc:439] T f411bbc269104667ae649f34867b9238 P 1d6b8fc0a2c44be9a8a3394663274a3a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc15feffe082457c99a95478676e263d" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46215 } } peers { permanent_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 46767 } } peers { permanent_uuid: "d0d9e88508924474886ee081ba8d6237" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 37001 } } }
I20260621 07:47:14.980700 2434 tablet_replica.cc:442] T f411bbc269104667ae649f34867b9238 P 1d6b8fc0a2c44be9a8a3394663274a3a: This TxnStatusTablet replica's current role is: LEADER
I20260621 07:47:14.981323 2400 tablet_replica.cc:439] T f411bbc269104667ae649f34867b9238 P d0d9e88508924474886ee081ba8d6237: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc15feffe082457c99a95478676e263d" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46215 } } peers { permanent_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 46767 } } peers { permanent_uuid: "d0d9e88508924474886ee081ba8d6237" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 37001 } } }
I20260621 07:47:14.981416 2400 tablet_replica.cc:442] T f411bbc269104667ae649f34867b9238 P d0d9e88508924474886ee081ba8d6237: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:47:14.985061 2434 tablet_replica.cc:439] T f411bbc269104667ae649f34867b9238 P 1d6b8fc0a2c44be9a8a3394663274a3a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc15feffe082457c99a95478676e263d" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46215 } } peers { permanent_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 46767 } } peers { permanent_uuid: "d0d9e88508924474886ee081ba8d6237" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 37001 } } }
I20260621 07:47:14.985168 2434 tablet_replica.cc:442] T f411bbc269104667ae649f34867b9238 P 1d6b8fc0a2c44be9a8a3394663274a3a: This TxnStatusTablet replica's current role is: LEADER
I20260621 07:47:14.986375 2402 tablet_replica.cc:439] T f411bbc269104667ae649f34867b9238 P dc15feffe082457c99a95478676e263d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc15feffe082457c99a95478676e263d" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46215 } } peers { permanent_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 46767 } } peers { permanent_uuid: "d0d9e88508924474886ee081ba8d6237" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 37001 } } }
I20260621 07:47:14.986479 2402 tablet_replica.cc:442] T f411bbc269104667ae649f34867b9238 P dc15feffe082457c99a95478676e263d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:47:14.989554 2387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:14.985254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41544 (local address 127.0.192.195:37001)
0621 07:47:14.985366 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:14.985370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:14.985403 (+ 33us) server_negotiation.cc:407] Connection header received
0621 07:47:14.985544 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:14.985547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:14.985605 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:14.985715 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:47:14.986079 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.986788 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.988206 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.988430 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.988548 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:14.988859 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:47:14.989198 (+ 339us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:47:14.989311 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:14.989399 (+ 88us) server_negotiation.cc:299] Negotiation successful
0621 07:47:14.989441 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260621 07:47:14.998960 2389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:14.995852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:55413 (local address 127.0.192.194:46767)
0621 07:47:14.995947 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:14.995950 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:14.996016 (+ 66us) server_negotiation.cc:407] Connection header received
0621 07:47:14.996106 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:14.996108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:14.996149 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:14.996211 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:47:14.996490 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.996973 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.998066 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.998256 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.998331 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:14.998625 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:47:14.998679 (+ 54us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:47:14.998737 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:14.998844 (+ 107us) server_negotiation.cc:299] Negotiation successful
0621 07:47:14.998881 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260621 07:47:14.999007 2375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:14.995774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:46767 (local address 127.0.192.195:55413)
0621 07:47:14.995951 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:14.995963 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:14.996042 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:14.996260 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:14.996263 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:14.996270 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 07:47:14.996397 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:14.996402 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.997076 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.997079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:14.997959 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:14.997968 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:14.998357 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:14.998360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:14.998457 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:14.998502 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 07:47:14.998761 (+ 259us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 07:47:14.998767 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 07:47:14.998828 (+ 61us) client_negotiation.cc:240] Negotiation successful
0621 07:47:14.998894 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260621 07:47:15.005599 2182 raft_consensus.cc:1261] T b24a9656f2214c09aaed4ab5a8bdd32e P 1d6b8fc0a2c44be9a8a3394663274a3a [term 1 FOLLOWER]: Refusing update from remote peer d0d9e88508924474886ee081ba8d6237: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:47:15.005599 2043 raft_consensus.cc:1261] T b24a9656f2214c09aaed4ab5a8bdd32e P dc15feffe082457c99a95478676e263d [term 1 FOLLOWER]: Refusing update from remote peer d0d9e88508924474886ee081ba8d6237: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:47:15.006067 2400 consensus_queue.cc:1035] T b24a9656f2214c09aaed4ab5a8bdd32e P d0d9e88508924474886ee081ba8d6237 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d6b8fc0a2c44be9a8a3394663274a3a" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:47:15.006247 2421 consensus_queue.cc:1035] T b24a9656f2214c09aaed4ab5a8bdd32e P d0d9e88508924474886ee081ba8d6237 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc15feffe082457c99a95478676e263d" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:47:15.028330 2387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:15.024837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:50251 (local address 127.0.192.195:37001)
0621 07:47:15.024950 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:15.024954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:15.025143 (+ 189us) server_negotiation.cc:407] Connection header received
0621 07:47:15.025231 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:15.025233 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:15.025265 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:15.025357 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:47:15.025606 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:15.026219 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:15.027436 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:15.027652 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:15.027735 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:15.028001 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:47:15.028061 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:47:15.028122 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:15.028222 (+ 100us) server_negotiation.cc:299] Negotiation successful
0621 07:47:15.028257 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260621 07:47:15.028477 2448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:15.024774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.195:37001 (local address 127.0.192.194:50251)
0621 07:47:15.025075 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:15.025086 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:15.025168 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:15.025363 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:15.025366 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:15.025372 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 07:47:15.025487 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:15.025492 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:15.026409 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:15.026412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:15.027305 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:15.027314 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:15.027765 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:15.027768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:15.027850 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:15.027891 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 07:47:15.028145 (+ 254us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 07:47:15.028151 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 07:47:15.028216 (+ 65us) client_negotiation.cc:240] Negotiation successful
0621 07:47:15.028276 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":129,"threads_started":1}
I20260621 07:47:15.049188 1884 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:47:15.049331 1884 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:47:15.051170 1884 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c4822e5a59714dac88935d524bd2299a: Sending DeleteTablet for 3 replicas of tablet b24a9656f2214c09aaed4ab5a8bdd32e
I20260621 07:47:15.051918 2303 tablet_service.cc:1514] Processing DeleteTablet for tablet b24a9656f2214c09aaed4ab5a8bdd32e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:47:15 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:41536
I20260621 07:47:15.052079 2414 tablet_replica.cc:330] T b24a9656f2214c09aaed4ab5a8bdd32e P d0d9e88508924474886ee081ba8d6237: stopping tablet replica
I20260621 07:47:15.052201 2414 raft_consensus.cc:2229] T b24a9656f2214c09aaed4ab5a8bdd32e P d0d9e88508924474886ee081ba8d6237 [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:47:15.052448 2414 raft_consensus.cc:2258] T b24a9656f2214c09aaed4ab5a8bdd32e P d0d9e88508924474886ee081ba8d6237 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:47:15.052778 2023 tablet_service.cc:1514] Processing DeleteTablet for tablet b24a9656f2214c09aaed4ab5a8bdd32e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:47:15 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:56422
I20260621 07:47:15.052976 2416 tablet_replica.cc:330] T b24a9656f2214c09aaed4ab5a8bdd32e P dc15feffe082457c99a95478676e263d: stopping tablet replica
I20260621 07:47:15.053103 2416 raft_consensus.cc:2229] T b24a9656f2214c09aaed4ab5a8bdd32e P dc15feffe082457c99a95478676e263d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:47:15.053123 2163 tablet_service.cc:1514] Processing DeleteTablet for tablet b24a9656f2214c09aaed4ab5a8bdd32e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:47:15 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:42166
I20260621 07:47:15.053164 2414 ts_tablet_manager.cc:1905] T b24a9656f2214c09aaed4ab5a8bdd32e P d0d9e88508924474886ee081ba8d6237: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:47:15.053210 2416 raft_consensus.cc:2258] T b24a9656f2214c09aaed4ab5a8bdd32e P dc15feffe082457c99a95478676e263d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:47:15.054098 2416 ts_tablet_manager.cc:1905] T b24a9656f2214c09aaed4ab5a8bdd32e P dc15feffe082457c99a95478676e263d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:47:15.055766 2414 ts_tablet_manager.cc:1918] T b24a9656f2214c09aaed4ab5a8bdd32e P d0d9e88508924474886ee081ba8d6237: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 07:47:15.055845 2414 log.cc:1196] T b24a9656f2214c09aaed4ab5a8bdd32e P d0d9e88508924474886ee081ba8d6237: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-2/wal/wals/b24a9656f2214c09aaed4ab5a8bdd32e
I20260621 07:47:15.056087 2414 ts_tablet_manager.cc:1939] T b24a9656f2214c09aaed4ab5a8bdd32e P d0d9e88508924474886ee081ba8d6237: Deleting consensus metadata
I20260621 07:47:15.056387 2416 ts_tablet_manager.cc:1918] T b24a9656f2214c09aaed4ab5a8bdd32e P dc15feffe082457c99a95478676e263d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 07:47:15.056478 2416 log.cc:1196] T b24a9656f2214c09aaed4ab5a8bdd32e P dc15feffe082457c99a95478676e263d: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-0/wal/wals/b24a9656f2214c09aaed4ab5a8bdd32e
I20260621 07:47:15.056674 2415 tablet_replica.cc:330] T b24a9656f2214c09aaed4ab5a8bdd32e P 1d6b8fc0a2c44be9a8a3394663274a3a: stopping tablet replica
I20260621 07:47:15.056784 2416 ts_tablet_manager.cc:1939] T b24a9656f2214c09aaed4ab5a8bdd32e P dc15feffe082457c99a95478676e263d: Deleting consensus metadata
I20260621 07:47:15.056754 1873 catalog_manager.cc:4841] TS d0d9e88508924474886ee081ba8d6237 (127.0.192.195:37001): tablet b24a9656f2214c09aaed4ab5a8bdd32e (table test-table [id=435cbcffa30e40e69b56a496a8547432]) successfully deleted
I20260621 07:47:15.056785 2415 raft_consensus.cc:2229] T b24a9656f2214c09aaed4ab5a8bdd32e P 1d6b8fc0a2c44be9a8a3394663274a3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:47:15.056882 2415 raft_consensus.cc:2258] T b24a9656f2214c09aaed4ab5a8bdd32e P 1d6b8fc0a2c44be9a8a3394663274a3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:47:15.057559 2415 ts_tablet_manager.cc:1905] T b24a9656f2214c09aaed4ab5a8bdd32e P 1d6b8fc0a2c44be9a8a3394663274a3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:47:15.057547 1873 catalog_manager.cc:4841] TS dc15feffe082457c99a95478676e263d (127.0.192.193:46215): tablet b24a9656f2214c09aaed4ab5a8bdd32e (table test-table [id=435cbcffa30e40e69b56a496a8547432]) successfully deleted
I20260621 07:47:15.059549 2415 ts_tablet_manager.cc:1918] T b24a9656f2214c09aaed4ab5a8bdd32e P 1d6b8fc0a2c44be9a8a3394663274a3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 07:47:15.059607 2415 log.cc:1196] T b24a9656f2214c09aaed4ab5a8bdd32e P 1d6b8fc0a2c44be9a8a3394663274a3a: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782028028522580-771-0/minicluster-data/ts-1/wal/wals/b24a9656f2214c09aaed4ab5a8bdd32e
I20260621 07:47:15.059794 2415 ts_tablet_manager.cc:1939] T b24a9656f2214c09aaed4ab5a8bdd32e P 1d6b8fc0a2c44be9a8a3394663274a3a: Deleting consensus metadata
I20260621 07:47:15.060308 1873 catalog_manager.cc:4841] TS 1d6b8fc0a2c44be9a8a3394663274a3a (127.0.192.194:46767): tablet b24a9656f2214c09aaed4ab5a8bdd32e (table test-table [id=435cbcffa30e40e69b56a496a8547432]) successfully deleted
Jun 21 07:47:15 dist-test-slave-lzx3 krb5kdc[1841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028034, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.192.193@KRBTEST.COM
I20260621 07:47:15.070611 2391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:15.053488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56434 (local address 127.0.192.193:46215)
0621 07:47:15.053600 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:15.053606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:15.060765 (+ 7159us) server_negotiation.cc:407] Connection header received
0621 07:47:15.060878 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:15.060881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:15.060925 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:15.061004 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:15.061677 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:15.062136 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:15.063151 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:15.063377 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:15.063504 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:15.067065 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:15.067074 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:15.067087 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:15.067114 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:15.068872 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:15.069279 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:15.069285 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:15.069286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:15.069366 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:15.069649 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:15.069653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:15.069655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:15.069880 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:15.070069 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:15.070308 (+ 239us) server_negotiation.cc:299] Negotiation successful
0621 07:47:15.070418 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260621 07:47:15.070969 2083 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56434
I20260621 07:47:15.079833 1935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:15.072592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57010 (local address 127.0.192.254:44477)
0621 07:47:15.072718 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:15.072723 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:15.072877 (+ 154us) server_negotiation.cc:407] Connection header received
0621 07:47:15.073017 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:15.073020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:15.073064 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:15.073169 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:15.073791 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:15.074275 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:15.075098 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:15.075307 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:15.075410 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:15.076720 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:15.076724 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:15.076726 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:15.076750 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:15.078189 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:15.078631 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:15.078635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:15.078637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:15.078717 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:15.079000 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:15.079004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:15.079005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:15.079178 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:15.079306 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:15.079543 (+ 237us) server_negotiation.cc:299] Negotiation successful
0621 07:47:15.079651 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260621 07:47:15.080183 1884 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57010
I20260621 07:47:15.081617 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 1958
I20260621 07:47:15.091183 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 2098
I20260621 07:47:15.097391 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 2238
I20260621 07:47:15.101591 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 1854
2026-06-21T07:47:15Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2555 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:47:15 dist-test-slave-lzx3 krb5kdc[2457](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:47:15 dist-test-slave-lzx3 krb5kdc[2457](info): set up 2 sockets
Jun 21 07:47:15 dist-test-slave-lzx3 krb5kdc[2457](info): commencing operation
krb5kdc: starting...
W20260621 07:47:16.064167 771 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/krb5kdc/test-user.keytab.
Jun 21 07:47:16 dist-test-slave-lzx3 krb5kdc[2457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:47:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:47:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.192.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
I20260621 07:47:16.165478 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:40179
--webserver_interface=127.0.192.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:46497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:40179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:16.259800 2470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:16.260054 2470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:16.260118 2470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:16.263765 2470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:47:16.263845 2470 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260621 07:47:16.263876 2470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:16.263914 2470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:16.263947 2470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:47:16.263979 2470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:47:16.268689 2470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:46497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:40179
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:40179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:16.269582 2470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:16.270442 2470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:16.277668 2475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:16.277721 2470 server_base.cc:1034] running on GCE node
W20260621 07:47:16.277693 2478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:16.277678 2476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:16.278486 2470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:16.279363 2470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:16.280535 2470 hybrid_clock.cc:648] HybridClock initialized: now 1782028036280520 us; error 27 us; skew 500 ppm
Jun 21 07:47:16 dist-test-slave-lzx3 krb5kdc[2457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:16.282987 2470 init.cc:377] Logged in from keytab as kudu/127.0.192.254@KRBTEST.COM (short username kudu)
I20260621 07:47:16.283759 2470 webserver.cc:460] Webserver started at http://127.0.192.254:45591/ using document root <none> and password file <none>
I20260621 07:47:16.284214 2470 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:16.284299 2470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:16.284514 2470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:16.286222 2470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "147c497bf1a945a1a37de938664b2f47"
format_stamp: "Formatted at 2026-06-21 07:47:16 on dist-test-slave-lzx3"
server_key: "cefde0a29e16ca5a9e12e21f0eb69fd5"
server_key_iv: "e7230dab312b9c8263b45a540f457f82"
server_key_version: "encryptionkey@0"
I20260621 07:47:16.286751 2470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "147c497bf1a945a1a37de938664b2f47"
format_stamp: "Formatted at 2026-06-21 07:47:16 on dist-test-slave-lzx3"
server_key: "cefde0a29e16ca5a9e12e21f0eb69fd5"
server_key_iv: "e7230dab312b9c8263b45a540f457f82"
server_key_version: "encryptionkey@0"
I20260621 07:47:16.290693 2470 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 07:47:16.292786 2485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:16.293848 2470 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:16.293977 2470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "147c497bf1a945a1a37de938664b2f47"
format_stamp: "Formatted at 2026-06-21 07:47:16 on dist-test-slave-lzx3"
server_key: "cefde0a29e16ca5a9e12e21f0eb69fd5"
server_key_iv: "e7230dab312b9c8263b45a540f457f82"
server_key_version: "encryptionkey@0"
I20260621 07:47:16.294077 2470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:16.304944 2470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:16.307629 2470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:16.307792 2470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:16.314678 2470 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:40179
I20260621 07:47:16.314741 2547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:40179 every 8 connection(s)
I20260621 07:47:16.315448 2470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:47:16.317826 2548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:16.319878 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 2470
I20260621 07:47:16.320022 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:47:16.320243 771 external_mini_cluster.cc:1428] Setting key e4d7ca88b43ce070b438c835249cb5ff
I20260621 07:47:16.323087 2548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 147c497bf1a945a1a37de938664b2f47: Bootstrap starting.
I20260621 07:47:16.325593 2548 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 147c497bf1a945a1a37de938664b2f47: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:16.326247 2548 log.cc:824] T 00000000000000000000000000000000 P 147c497bf1a945a1a37de938664b2f47: Log is configured to *not* fsync() on all Append() calls
I20260621 07:47:16.327853 2548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 147c497bf1a945a1a37de938664b2f47: No bootstrap required, opened a new log
Jun 21 07:47:16 dist-test-slave-lzx3 krb5kdc[2457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:47:16.331023 2548 raft_consensus.cc:348] T 00000000000000000000000000000000 P 147c497bf1a945a1a37de938664b2f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147c497bf1a945a1a37de938664b2f47" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40179 } }
I20260621 07:47:16.331243 2548 raft_consensus.cc:374] T 00000000000000000000000000000000 P 147c497bf1a945a1a37de938664b2f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:16.331300 2548 raft_consensus.cc:729] T 00000000000000000000000000000000 P 147c497bf1a945a1a37de938664b2f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 147c497bf1a945a1a37de938664b2f47, State: Initialized, Role: FOLLOWER
I20260621 07:47:16.331693 2548 consensus_queue.cc:260] T 00000000000000000000000000000000 P 147c497bf1a945a1a37de938664b2f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147c497bf1a945a1a37de938664b2f47" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40179 } }
I20260621 07:47:16.331859 2548 raft_consensus.cc:388] T 00000000000000000000000000000000 P 147c497bf1a945a1a37de938664b2f47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:47:16.331924 2548 raft_consensus.cc:482] T 00000000000000000000000000000000 P 147c497bf1a945a1a37de938664b2f47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:47:16.332069 2548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 147c497bf1a945a1a37de938664b2f47 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:16.333065 2548 raft_consensus.cc:504] T 00000000000000000000000000000000 P 147c497bf1a945a1a37de938664b2f47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147c497bf1a945a1a37de938664b2f47" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40179 } }
I20260621 07:47:16.333408 2548 leader_election.cc:302] T 00000000000000000000000000000000 P 147c497bf1a945a1a37de938664b2f47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 147c497bf1a945a1a37de938664b2f47; no voters:
I20260621 07:47:16.333700 2548 leader_election.cc:288] T 00000000000000000000000000000000 P 147c497bf1a945a1a37de938664b2f47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:47:16.333801 2553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 147c497bf1a945a1a37de938664b2f47 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:47:16.333986 2553 raft_consensus.cc:686] T 00000000000000000000000000000000 P 147c497bf1a945a1a37de938664b2f47 [term 1 LEADER]: Becoming Leader. State: Replica: 147c497bf1a945a1a37de938664b2f47, State: Running, Role: LEADER
I20260621 07:47:16.334131 2551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:16.321392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58228 (local address 127.0.192.254:40179)
0621 07:47:16.321776 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:16.321793 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:16.321871 (+ 78us) server_negotiation.cc:407] Connection header received
0621 07:47:16.322726 (+ 855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:16.322746 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:16.323111 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:16.323456 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:16.324081 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:16.325604 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:16.326452 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:16.326916 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:16.327085 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:16.329539 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:16.329543 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:16.329552 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:16.329574 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:16.331117 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:16.331586 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:16.331591 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:16.331595 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:16.331706 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:16.332049 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:16.332054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:16.332056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:16.332488 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:16.332716 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:16.332973 (+ 257us) server_negotiation.cc:299] Negotiation successful
0621 07:47:16.333244 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":90,"threads_started":1}
I20260621 07:47:16.334292 2553 consensus_queue.cc:237] T 00000000000000000000000000000000 P 147c497bf1a945a1a37de938664b2f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147c497bf1a945a1a37de938664b2f47" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40179 } }
I20260621 07:47:16.334743 2548 sys_catalog.cc:564] T 00000000000000000000000000000000 P 147c497bf1a945a1a37de938664b2f47 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:47:16.335947 2555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 147c497bf1a945a1a37de938664b2f47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 147c497bf1a945a1a37de938664b2f47. Latest consensus state: current_term: 1 leader_uuid: "147c497bf1a945a1a37de938664b2f47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147c497bf1a945a1a37de938664b2f47" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40179 } } }
I20260621 07:47:16.336058 2555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 147c497bf1a945a1a37de938664b2f47 [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:16.336421 2564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:47:16.336951 2554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 147c497bf1a945a1a37de938664b2f47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "147c497bf1a945a1a37de938664b2f47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147c497bf1a945a1a37de938664b2f47" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40179 } } }
I20260621 07:47:16.337041 2554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 147c497bf1a945a1a37de938664b2f47 [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:16.339064 2564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:47:16.343776 2564 catalog_manager.cc:1347] Generated new cluster ID: 722c881c99da472aa77bdc3a9e6dc208
I20260621 07:47:16.343830 2564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:47:16.352397 2564 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:47:16.353245 2564 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:47:16.358707 2564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 147c497bf1a945a1a37de938664b2f47: Generated new TSK 0
I20260621 07:47:16.359238 2564 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.192.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab.
Entry for principal kudu/127.0.192.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab.
I20260621 07:47:16.393106 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.193:0
--local_ip_for_outbound_sockets=127.0.192.193
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:40179
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:46497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:16.496507 2574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:16.496737 2574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:16.496793 2574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:16.500918 2574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:16.500972 2574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:16.501012 2574 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260621 07:47:16.501026 2574 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 07:47:16.501057 2574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.193
I20260621 07:47:16.506181 2574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:46497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.192.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.192.254:40179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.193
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:16.507009 2574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:16.507934 2574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:16.516860 2582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:16.516870 2579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:16.516876 2580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:16.516953 2574 server_base.cc:1034] running on GCE node
I20260621 07:47:16.517493 2574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:16.518000 2574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:16.519207 2574 hybrid_clock.cc:648] HybridClock initialized: now 1782028036519192 us; error 33 us; skew 500 ppm
Jun 21 07:47:16 dist-test-slave-lzx3 krb5kdc[2457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:16.521898 2574 init.cc:377] Logged in from keytab as kudu/127.0.192.193@KRBTEST.COM (short username kudu)
I20260621 07:47:16.522702 2574 webserver.cc:460] Webserver started at http://127.0.192.193:34535/ using document root <none> and password file <none>
I20260621 07:47:16.523228 2574 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:16.523331 2574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:16.523562 2574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:16.525502 2574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-0/data/instance:
uuid: "1750da561edc402883853661828d10c6"
format_stamp: "Formatted at 2026-06-21 07:47:16 on dist-test-slave-lzx3"
server_key: "8fa91b9e69a8bc29388bd5b65e5cbd8a"
server_key_iv: "796c5b4edc28582de1648e97cbbd7202"
server_key_version: "encryptionkey@0"
I20260621 07:47:16.526062 2574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-0/wal/instance:
uuid: "1750da561edc402883853661828d10c6"
format_stamp: "Formatted at 2026-06-21 07:47:16 on dist-test-slave-lzx3"
server_key: "8fa91b9e69a8bc29388bd5b65e5cbd8a"
server_key_iv: "796c5b4edc28582de1648e97cbbd7202"
server_key_version: "encryptionkey@0"
I20260621 07:47:16.529415 2574 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260621 07:47:16.531733 2589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:16.532887 2574 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260621 07:47:16.533015 2574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-0/wal
uuid: "1750da561edc402883853661828d10c6"
format_stamp: "Formatted at 2026-06-21 07:47:16 on dist-test-slave-lzx3"
server_key: "8fa91b9e69a8bc29388bd5b65e5cbd8a"
server_key_iv: "796c5b4edc28582de1648e97cbbd7202"
server_key_version: "encryptionkey@0"
I20260621 07:47:16.533120 2574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:16.549468 2574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:16.551945 2574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:16.552175 2574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:16.553511 2574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:47:16.553591 2574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:16.553655 2574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:47:16.553714 2574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 07:47:16 dist-test-slave-lzx3 krb5kdc[2457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:47:16.563988 2574 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.193:45443
I20260621 07:47:16.564284 2704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.193:45443 every 8 connection(s)
I20260621 07:47:16.565069 2574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
I20260621 07:47:16.566992 2551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:16.554338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:46339 (local address 127.0.192.254:40179)
0621 07:47:16.554471 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:16.554474 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:16.555340 (+ 866us) server_negotiation.cc:407] Connection header received
0621 07:47:16.556279 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:16.556285 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:16.556366 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:16.556470 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:16.557682 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:16.558311 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:16.559941 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:16.560193 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:16.560285 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:16.563056 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:16.563061 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:16.563064 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:16.563093 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:16.564862 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:16.565415 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:16.565418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:16.565420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:16.565517 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:16.565933 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:16.565937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:16.565939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:16.566113 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:16.566227 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:16.566700 (+ 473us) server_negotiation.cc:299] Negotiation successful
0621 07:47:16.566812 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 07:47:16.567696 2598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:16.554791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:40179 (local address 127.0.192.193:46339)
0621 07:47:16.555203 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:16.555237 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:16.556042 (+ 805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:16.556771 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:16.556781 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:16.557079 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:16.557542 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:16.557555 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:16.558462 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:16.558470 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:16.559792 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:16.559802 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:16.560416 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:16.560419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:16.560566 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:16.561141 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:16.561160 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:16.562874 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:16.564987 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:16.564991 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:16.565001 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:16.565310 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:16.565635 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:16.565639 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:16.565643 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:16.565824 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:16.566237 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:16.566245 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:16.566466 (+ 221us) client_negotiation.cc:769] Sending connection context
0621 07:47:16.566647 (+ 181us) client_negotiation.cc:240] Negotiation successful
0621 07:47:16.566927 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":118,"threads_started":1}
I20260621 07:47:16.568126 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 2574
I20260621 07:47:16.568259 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-0/wal/instance
I20260621 07:47:16.568539 771 external_mini_cluster.cc:1428] Setting key a58331b44382960312a1ff9c747697a0
I20260621 07:47:16.570567 2705 heartbeater.cc:347] Connected to a master server at 127.0.192.254:40179
I20260621 07:47:16.570701 2705 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:16.571130 2705 heartbeater.cc:511] Master 127.0.192.254:40179 requested a full tablet report, sending...
I20260621 07:47:16.572535 2496 ts_manager.cc:194] Registered new tserver with Master: 1750da561edc402883853661828d10c6 (127.0.192.193:45443)
I20260621 07:47:16.573939 2496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.193@KRBTEST.COM'} at 127.0.192.193:46339
I20260621 07:47:16.578136 2551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:16.570771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:38823 (local address 127.0.192.254:40179)
0621 07:47:16.570866 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:16.570869 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:16.571108 (+ 239us) server_negotiation.cc:407] Connection header received
0621 07:47:16.571258 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:16.571261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:16.571297 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:16.571338 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:16.571875 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:16.572636 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:16.573780 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:16.574004 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:16.574102 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:16.575131 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:16.575134 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:16.575137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:16.575157 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:16.576520 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:16.576988 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:16.576993 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:16.576994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:16.577073 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:16.577363 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:16.577367 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:16.577369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:16.577538 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:16.577644 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:16.577904 (+ 260us) server_negotiation.cc:299] Negotiation successful
0621 07:47:16.578002 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260621 07:47:16.578147 2711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:16.570791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:40179 (local address 127.0.192.193:38823)
0621 07:47:16.571037 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:16.571050 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:16.571161 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:16.571446 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:16.571448 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:16.571630 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:16.571759 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:16.571766 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:16.572794 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:16.572797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:16.573629 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:16.573639 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:16.574199 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:16.574201 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:16.574266 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:16.574715 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:16.574727 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:16.575003 (+ 276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:16.576674 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:16.576677 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:16.576679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:16.576881 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:16.577172 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:16.577177 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:16.577179 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:16.577265 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:16.577645 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:16.577649 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:16.577736 (+ 87us) client_negotiation.cc:769] Sending connection context
0621 07:47:16.577853 (+ 117us) client_negotiation.cc:240] Negotiation successful
0621 07:47:16.577975 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":97,"threads_started":1}
WARNING: no policy specified for kudu/127.0.192.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab.
Entry for principal kudu/127.0.192.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab.
I20260621 07:47:16.597322 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.194:0
--local_ip_for_outbound_sockets=127.0.192.194
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:40179
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:46497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:16.694763 2714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:16.694957 2714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:16.694998 2714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:16.698541 2714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:16.698582 2714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:16.698614 2714 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260621 07:47:16.698626 2714 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 07:47:16.698670 2714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.194
I20260621 07:47:16.702806 2714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:46497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.192.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.192.254:40179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.194
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:16.703538 2714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:16.704304 2714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:16.711946 2722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:16.711983 2719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:16.711992 2720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:16.712265 2714 server_base.cc:1034] running on GCE node
I20260621 07:47:16.712695 2714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:16.713222 2714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:16.714416 2714 hybrid_clock.cc:648] HybridClock initialized: now 1782028036714389 us; error 44 us; skew 500 ppm
Jun 21 07:47:16 dist-test-slave-lzx3 krb5kdc[2457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:16.716779 2714 init.cc:377] Logged in from keytab as kudu/127.0.192.194@KRBTEST.COM (short username kudu)
I20260621 07:47:16.717464 2714 webserver.cc:460] Webserver started at http://127.0.192.194:43631/ using document root <none> and password file <none>
I20260621 07:47:16.717902 2714 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:16.717983 2714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:16.718166 2714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:16.719815 2714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-1/data/instance:
uuid: "781c0beb96e8435a9dc711b0b70eb1da"
format_stamp: "Formatted at 2026-06-21 07:47:16 on dist-test-slave-lzx3"
server_key: "a1c872385b36ac08c167fad137cddeba"
server_key_iv: "1618749ffa2ab3a5fb1c4412c2fa1d27"
server_key_version: "encryptionkey@0"
I20260621 07:47:16.720343 2714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-1/wal/instance:
uuid: "781c0beb96e8435a9dc711b0b70eb1da"
format_stamp: "Formatted at 2026-06-21 07:47:16 on dist-test-slave-lzx3"
server_key: "a1c872385b36ac08c167fad137cddeba"
server_key_iv: "1618749ffa2ab3a5fb1c4412c2fa1d27"
server_key_version: "encryptionkey@0"
I20260621 07:47:16.723569 2714 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260621 07:47:16.725600 2729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:16.726636 2714 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 07:47:16.726753 2714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-1/wal
uuid: "781c0beb96e8435a9dc711b0b70eb1da"
format_stamp: "Formatted at 2026-06-21 07:47:16 on dist-test-slave-lzx3"
server_key: "a1c872385b36ac08c167fad137cddeba"
server_key_iv: "1618749ffa2ab3a5fb1c4412c2fa1d27"
server_key_version: "encryptionkey@0"
I20260621 07:47:16.726847 2714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:16.736819 2714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:16.739390 2714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:16.739554 2714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:16.740728 2714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:47:16.740775 2714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:16.740829 2714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:47:16.740860 2714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 07:47:16 dist-test-slave-lzx3 krb5kdc[2457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:47:16.750423 2714 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.194:40397
I20260621 07:47:16.750540 2844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.194:40397 every 8 connection(s)
I20260621 07:47:16.751199 2714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
I20260621 07:47:16.752365 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 2714
I20260621 07:47:16.752491 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-1/wal/instance
I20260621 07:47:16.752816 771 external_mini_cluster.cc:1428] Setting key 8be25812711c8622eb4dd0fb1de7f490
I20260621 07:47:16.753125 2551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:16.741340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:55723 (local address 127.0.192.254:40179)
0621 07:47:16.741459 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:16.741463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:16.742262 (+ 799us) server_negotiation.cc:407] Connection header received
0621 07:47:16.743067 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:16.743070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:16.743126 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:16.743234 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:16.744416 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:16.745066 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:16.746118 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:16.746425 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:16.746540 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:16.749587 (+ 3047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:16.749591 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:16.749593 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:16.749617 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:16.751004 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:16.751507 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:16.751510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:16.751512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:16.751598 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:16.751944 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:16.751947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:16.751949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:16.752189 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:16.752267 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:16.752903 (+ 636us) server_negotiation.cc:299] Negotiation successful
0621 07:47:16.753016 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 07:47:16.753969 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:16.741704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:40179 (local address 127.0.192.194:55723)
0621 07:47:16.742143 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:16.742176 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:16.742872 (+ 696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:16.743387 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:16.743395 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:16.743859 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:16.744284 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:16.744295 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:16.745185 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:16.745188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:16.746001 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:16.746009 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:16.746569 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:16.746573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:16.746777 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:16.747303 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:16.747326 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:16.749423 (+ 2097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:16.751108 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:16.751112 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:16.751123 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:16.751392 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:16.751697 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:16.751699 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:16.751701 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:16.751843 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:16.752303 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:16.752310 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:16.752626 (+ 316us) client_negotiation.cc:769] Sending connection context
0621 07:47:16.752841 (+ 215us) client_negotiation.cc:240] Negotiation successful
0621 07:47:16.753126 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":109,"threads_started":1}
I20260621 07:47:16.755215 2845 heartbeater.cc:347] Connected to a master server at 127.0.192.254:40179
I20260621 07:47:16.755326 2845 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:16.755760 2845 heartbeater.cc:511] Master 127.0.192.254:40179 requested a full tablet report, sending...
I20260621 07:47:16.756697 2496 ts_manager.cc:194] Registered new tserver with Master: 781c0beb96e8435a9dc711b0b70eb1da (127.0.192.194:40397)
I20260621 07:47:16.757454 2496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.194@KRBTEST.COM'} at 127.0.192.194:55723
WARNING: no policy specified for kudu/127.0.192.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.195@KRBTEST.COM" created.
I20260621 07:47:16.764868 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:16.757919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:40179 (local address 127.0.192.194:40869)
0621 07:47:16.758183 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:16.758196 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:16.758295 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:16.758587 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:16.758590 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:16.758775 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:16.758922 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:16.758929 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:16.760064 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:16.760066 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:16.760630 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:16.760638 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:16.761172 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:16.761174 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:16.761233 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:16.761677 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:16.761687 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:16.761943 (+ 256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:16.763442 (+ 1499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:16.763444 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:16.763446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:16.763614 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:16.763948 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:16.763953 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:16.763955 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:16.764034 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:16.764382 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:16.764384 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:16.764443 (+ 59us) client_negotiation.cc:769] Sending connection context
0621 07:47:16.764513 (+ 70us) client_negotiation.cc:240] Negotiation successful
0621 07:47:16.764662 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":9,"thread_start_us":108,"threads_started":1}
I20260621 07:47:16.765121 2551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:16.757938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:40869 (local address 127.0.192.254:40179)
0621 07:47:16.758079 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:16.758085 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:16.758284 (+ 199us) server_negotiation.cc:407] Connection header received
0621 07:47:16.758408 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:16.758412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:16.758466 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:16.758553 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:16.759059 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:16.759933 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:16.760730 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:16.761057 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:16.761169 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:16.762078 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:16.762084 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:16.762089 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:16.762120 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:16.763346 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:16.763700 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:16.763702 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:16.763703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:16.763830 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:16.764113 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:16.764115 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:16.764116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:16.764286 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:16.764374 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:16.764636 (+ 262us) server_negotiation.cc:299] Negotiation successful
0621 07:47:16.764747 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab.
Entry for principal kudu/127.0.192.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab.
I20260621 07:47:16.779350 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.195:0
--local_ip_for_outbound_sockets=127.0.192.195
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:40179
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:46497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:16.873059 2854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:16.873294 2854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:16.873373 2854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:16.876755 2854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:16.876839 2854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:16.876909 2854 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260621 07:47:16.876938 2854 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 07:47:16.876987 2854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.195
I20260621 07:47:16.881276 2854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:46497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.192.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.192.254:40179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.195
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:16.882035 2854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:16.882862 2854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:16.890967 2860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:16.890975 2862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:16.891032 2854 server_base.cc:1034] running on GCE node
W20260621 07:47:16.890975 2859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:16.891564 2854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:16.892129 2854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:16.893303 2854 hybrid_clock.cc:648] HybridClock initialized: now 1782028036893267 us; error 28 us; skew 500 ppm
Jun 21 07:47:16 dist-test-slave-lzx3 krb5kdc[2457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:16.895913 2854 init.cc:377] Logged in from keytab as kudu/127.0.192.195@KRBTEST.COM (short username kudu)
I20260621 07:47:16.896751 2854 webserver.cc:460] Webserver started at http://127.0.192.195:39821/ using document root <none> and password file <none>
I20260621 07:47:16.897248 2854 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:16.897334 2854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:16.897504 2854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:16.899287 2854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-2/data/instance:
uuid: "1cd096c3570149679c0417ccd70dfe74"
format_stamp: "Formatted at 2026-06-21 07:47:16 on dist-test-slave-lzx3"
server_key: "91c86bd5c6ee1190c0fe27ce1c794167"
server_key_iv: "022291bcf01ec54a38c478ff08f21dcd"
server_key_version: "encryptionkey@0"
I20260621 07:47:16.899822 2854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-2/wal/instance:
uuid: "1cd096c3570149679c0417ccd70dfe74"
format_stamp: "Formatted at 2026-06-21 07:47:16 on dist-test-slave-lzx3"
server_key: "91c86bd5c6ee1190c0fe27ce1c794167"
server_key_iv: "022291bcf01ec54a38c478ff08f21dcd"
server_key_version: "encryptionkey@0"
I20260621 07:47:16.903079 2854 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260621 07:47:16.905268 2869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:16.906260 2854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:16.906386 2854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-2/wal
uuid: "1cd096c3570149679c0417ccd70dfe74"
format_stamp: "Formatted at 2026-06-21 07:47:16 on dist-test-slave-lzx3"
server_key: "91c86bd5c6ee1190c0fe27ce1c794167"
server_key_iv: "022291bcf01ec54a38c478ff08f21dcd"
server_key_version: "encryptionkey@0"
I20260621 07:47:16.906492 2854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:16.921804 2854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:16.924844 2854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:16.925007 2854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:16.926151 2854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:47:16.926224 2854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:16.926283 2854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:47:16.926316 2854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 07:47:16 dist-test-slave-lzx3 krb5kdc[2457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:47:16.936266 2854 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.195:36063
I20260621 07:47:16.936599 2984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.195:36063 every 8 connection(s)
I20260621 07:47:16.937096 2854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
I20260621 07:47:16.938841 2551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:16.927012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:34443 (local address 127.0.192.254:40179)
0621 07:47:16.927125 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:16.927129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:16.928002 (+ 873us) server_negotiation.cc:407] Connection header received
0621 07:47:16.928964 (+ 962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:16.928967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:16.929020 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:16.929100 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:16.930404 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:16.931249 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:16.932387 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:16.932745 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:16.932876 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:16.935418 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:16.935436 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:16.935438 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:16.935457 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:16.937028 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:16.937570 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:16.937575 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:16.937577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:16.937666 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:16.938037 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:16.938039 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:16.938041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:16.938203 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:16.938281 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:16.938648 (+ 367us) server_negotiation.cc:299] Negotiation successful
0621 07:47:16.938733 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260621 07:47:16.939504 2878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:16.927446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:40179 (local address 127.0.192.195:34443)
0621 07:47:16.927850 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:16.927883 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:16.928778 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:16.929274 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:16.929281 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:16.929692 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:16.930231 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:16.930249 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:16.931376 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:16.931380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:16.932251 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:16.932260 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:16.932913 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:16.932918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:16.933080 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:16.933588 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:16.933606 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:16.935268 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:16.937139 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:16.937144 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:16.937155 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:16.937448 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:16.937777 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:16.937780 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:16.937782 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:16.937936 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:16.938278 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:16.938282 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:16.938461 (+ 179us) client_negotiation.cc:769] Sending connection context
0621 07:47:16.938596 (+ 135us) client_negotiation.cc:240] Negotiation successful
0621 07:47:16.938789 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":114,"threads_started":1}
I20260621 07:47:16.940479 2985 heartbeater.cc:347] Connected to a master server at 127.0.192.254:40179
I20260621 07:47:16.940630 2985 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:16.941044 2985 heartbeater.cc:511] Master 127.0.192.254:40179 requested a full tablet report, sending...
I20260621 07:47:16.941911 2502 ts_manager.cc:194] Registered new tserver with Master: 1cd096c3570149679c0417ccd70dfe74 (127.0.192.195:36063)
I20260621 07:47:16.942580 2502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.195@KRBTEST.COM'} at 127.0.192.195:34443
I20260621 07:47:16.944113 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 2854
I20260621 07:47:16.944213 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-2/wal/instance
I20260621 07:47:16.944459 771 external_mini_cluster.cc:1428] Setting key bbe241ffecc43bbaead40de436536b4d
I20260621 07:47:16.946534 771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:47:16.949206 2991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:16.942254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:40179 (local address 127.0.192.195:45729)
0621 07:47:16.942528 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:16.942538 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:16.942639 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:16.942901 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:16.942904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:16.943084 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:16.943165 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:16.943170 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:16.943906 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:16.943909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:16.944703 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:16.944730 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:16.945284 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:16.945287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:16.945378 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:16.945812 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:16.945828 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:16.946109 (+ 281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:16.947740 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:16.947746 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:16.947750 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:16.948003 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:16.948350 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:16.948353 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:16.948355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:16.948433 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:16.948881 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:16.948883 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:16.948944 (+ 61us) client_negotiation.cc:769] Sending connection context
0621 07:47:16.949021 (+ 77us) client_negotiation.cc:240] Negotiation successful
0621 07:47:16.949103 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":90,"threads_started":1}
I20260621 07:47:16.949396 2551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:16.942321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:45729 (local address 127.0.192.254:40179)
0621 07:47:16.942414 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:16.942417 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:16.942606 (+ 189us) server_negotiation.cc:407] Connection header received
0621 07:47:16.942750 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:16.942754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:16.942792 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:16.942886 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:16.943281 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:16.943759 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:16.944859 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:16.945163 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:16.945268 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:16.946239 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:16.946242 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:16.946246 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:16.946269 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:16.947595 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:16.948115 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:16.948118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:16.948131 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:16.948238 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:16.948549 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:16.948554 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:16.948571 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:16.948788 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:16.948891 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:16.949100 (+ 209us) server_negotiation.cc:299] Negotiation successful
0621 07:47:16.949201 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Jun 21 07:47:16 dist-test-slave-lzx3 krb5kdc[2457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 07:47:16 dist-test-slave-lzx3 krb5kdc[2457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:47:16.970497 2551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:16.962877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58242 (local address 127.0.192.254:40179)
0621 07:47:16.963013 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:16.963016 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:16.963027 (+ 11us) server_negotiation.cc:407] Connection header received
0621 07:47:16.963055 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:16.963057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:16.963086 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:16.963177 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:16.963838 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:16.964286 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:16.965196 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:16.965411 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:16.965527 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:16.967650 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:16.967652 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:16.967654 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:16.967672 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:16.968964 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:16.969333 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:16.969337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:16.969339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:16.969413 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:16.969712 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:16.969716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:16.969718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:16.969866 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:16.969992 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:16.970230 (+ 238us) server_negotiation.cc:299] Negotiation successful
0621 07:47:16.970331 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 07:47:16.973127 2502 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58242:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:47:16.975854 2502 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:47:16.988077 3006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:16.983618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58144 (local address 127.0.192.193:45443)
0621 07:47:16.984054 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:16.984065 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:16.984084 (+ 19us) server_negotiation.cc:407] Connection header received
0621 07:47:16.984152 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:16.984156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:16.984340 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:16.984702 (+ 362us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:16.984814 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:16.985609 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:16.986761 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:16.987439 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:16.987557 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:16.987701 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:16.987855 (+ 154us) server_negotiation.cc:299] Negotiation successful
0621 07:47:16.987942 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":90,"threads_started":1}
I20260621 07:47:16.988128 3003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:16.982869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:40397 (local address 127.0.0.1:58982)
0621 07:47:16.983301 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:16.983335 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:16.983434 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:16.984234 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:16.984252 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:16.984273 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:16.984395 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:16.984405 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:16.985601 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:16.985604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:16.986749 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:16.986758 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:16.987734 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:16.987737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:16.987820 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:16.987841 (+ 21us) client_negotiation.cc:769] Sending connection context
0621 07:47:16.987886 (+ 45us) client_negotiation.cc:240] Negotiation successful
0621 07:47:16.987960 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":67,"thread_start_us":68,"threads_started":1}
I20260621 07:47:16.988137 3002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:16.982962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:45443 (local address 127.0.0.1:58144)
0621 07:47:16.983788 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:16.983818 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:16.983915 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:16.984471 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:16.984473 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:16.984485 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:16.984564 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:16.984570 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:16.985722 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:16.985725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:16.986627 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:16.986636 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:16.987568 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:16.987571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:16.987662 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:16.987697 (+ 35us) client_negotiation.cc:769] Sending connection context
0621 07:47:16.987825 (+ 128us) client_negotiation.cc:240] Negotiation successful
0621 07:47:16.987945 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"mutex_wait_us":6,"thread_start_us":68,"threads_started":1}
I20260621 07:47:16.988583 3005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:16.983533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.195:36063 (local address 127.0.0.1:51408)
0621 07:47:16.984227 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:16.984251 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:16.984387 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:16.984894 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:16.984896 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:16.984910 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:16.984986 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:16.984991 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:16.986193 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:16.986196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:16.987166 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:16.987174 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:16.988231 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:16.988236 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:16.988312 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:16.988333 (+ 21us) client_negotiation.cc:769] Sending connection context
0621 07:47:16.988399 (+ 66us) client_negotiation.cc:240] Negotiation successful
0621 07:47:16.988455 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":76,"threads_started":1}
I20260621 07:47:16.988852 3004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:16.983003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58982 (local address 127.0.192.194:40397)
0621 07:47:16.983741 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:16.983755 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:16.983781 (+ 26us) server_negotiation.cc:407] Connection header received
0621 07:47:16.983872 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:16.983891 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:16.984076 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:16.984530 (+ 454us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:16.984668 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:16.985488 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:16.986873 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:16.987614 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:16.988436 (+ 822us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:16.988591 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:16.988638 (+ 47us) server_negotiation.cc:299] Negotiation successful
0621 07:47:16.988731 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":105,"threads_started":1}
I20260621 07:47:16.988977 3007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:16.983671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51408 (local address 127.0.192.195:36063)
0621 07:47:16.984121 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:16.984130 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:16.984310 (+ 180us) server_negotiation.cc:407] Connection header received
0621 07:47:16.984506 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:16.984510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:16.984682 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:16.984767 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:16.985103 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:16.986070 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:16.987280 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:16.987968 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:16.988051 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:16.988178 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:16.988763 (+ 585us) server_negotiation.cc:299] Negotiation successful
0621 07:47:16.988846 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":94,"threads_started":1}
I20260621 07:47:16.991039 2779 tablet_service.cc:1467] Processing CreateTablet for tablet e22fb9a57eca4954848993c2abfdfa6a (DEFAULT_TABLE table=test-table [id=d69123da9e894953b123c26f58c206cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:47:16.990953 2919 tablet_service.cc:1467] Processing CreateTablet for tablet e22fb9a57eca4954848993c2abfdfa6a (DEFAULT_TABLE table=test-table [id=d69123da9e894953b123c26f58c206cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:47:16.990953 2639 tablet_service.cc:1467] Processing CreateTablet for tablet e22fb9a57eca4954848993c2abfdfa6a (DEFAULT_TABLE table=test-table [id=d69123da9e894953b123c26f58c206cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:47:16.991832 2779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e22fb9a57eca4954848993c2abfdfa6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:16.991861 2639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e22fb9a57eca4954848993c2abfdfa6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:16.991878 2919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e22fb9a57eca4954848993c2abfdfa6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:16.998677 3008 tablet_bootstrap.cc:492] T e22fb9a57eca4954848993c2abfdfa6a P 781c0beb96e8435a9dc711b0b70eb1da: Bootstrap starting.
I20260621 07:47:16.999490 3009 tablet_bootstrap.cc:492] T e22fb9a57eca4954848993c2abfdfa6a P 1cd096c3570149679c0417ccd70dfe74: Bootstrap starting.
I20260621 07:47:17.000756 3010 tablet_bootstrap.cc:492] T e22fb9a57eca4954848993c2abfdfa6a P 1750da561edc402883853661828d10c6: Bootstrap starting.
I20260621 07:47:17.000804 3008 tablet_bootstrap.cc:654] T e22fb9a57eca4954848993c2abfdfa6a P 781c0beb96e8435a9dc711b0b70eb1da: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:17.001410 3009 tablet_bootstrap.cc:654] T e22fb9a57eca4954848993c2abfdfa6a P 1cd096c3570149679c0417ccd70dfe74: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:17.001575 3008 log.cc:824] T e22fb9a57eca4954848993c2abfdfa6a P 781c0beb96e8435a9dc711b0b70eb1da: Log is configured to *not* fsync() on all Append() calls
I20260621 07:47:17.002100 3009 log.cc:824] T e22fb9a57eca4954848993c2abfdfa6a P 1cd096c3570149679c0417ccd70dfe74: Log is configured to *not* fsync() on all Append() calls
I20260621 07:47:17.002416 3010 tablet_bootstrap.cc:654] T e22fb9a57eca4954848993c2abfdfa6a P 1750da561edc402883853661828d10c6: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:17.003153 3010 log.cc:824] T e22fb9a57eca4954848993c2abfdfa6a P 1750da561edc402883853661828d10c6: Log is configured to *not* fsync() on all Append() calls
I20260621 07:47:17.003485 3008 tablet_bootstrap.cc:492] T e22fb9a57eca4954848993c2abfdfa6a P 781c0beb96e8435a9dc711b0b70eb1da: No bootstrap required, opened a new log
I20260621 07:47:17.003799 3008 ts_tablet_manager.cc:1397] T e22fb9a57eca4954848993c2abfdfa6a P 781c0beb96e8435a9dc711b0b70eb1da: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260621 07:47:17.004035 3009 tablet_bootstrap.cc:492] T e22fb9a57eca4954848993c2abfdfa6a P 1cd096c3570149679c0417ccd70dfe74: No bootstrap required, opened a new log
I20260621 07:47:17.004192 3009 ts_tablet_manager.cc:1397] T e22fb9a57eca4954848993c2abfdfa6a P 1cd096c3570149679c0417ccd70dfe74: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260621 07:47:17.004894 3010 tablet_bootstrap.cc:492] T e22fb9a57eca4954848993c2abfdfa6a P 1750da561edc402883853661828d10c6: No bootstrap required, opened a new log
I20260621 07:47:17.005062 3010 ts_tablet_manager.cc:1397] T e22fb9a57eca4954848993c2abfdfa6a P 1750da561edc402883853661828d10c6: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260621 07:47:17.007375 3010 raft_consensus.cc:348] T e22fb9a57eca4954848993c2abfdfa6a P 1750da561edc402883853661828d10c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } }
I20260621 07:47:17.007579 3010 raft_consensus.cc:374] T e22fb9a57eca4954848993c2abfdfa6a P 1750da561edc402883853661828d10c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:17.007455 3009 raft_consensus.cc:348] T e22fb9a57eca4954848993c2abfdfa6a P 1cd096c3570149679c0417ccd70dfe74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } }
I20260621 07:47:17.007455 3008 raft_consensus.cc:348] T e22fb9a57eca4954848993c2abfdfa6a P 781c0beb96e8435a9dc711b0b70eb1da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } }
I20260621 07:47:17.007625 3010 raft_consensus.cc:729] T e22fb9a57eca4954848993c2abfdfa6a P 1750da561edc402883853661828d10c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1750da561edc402883853661828d10c6, State: Initialized, Role: FOLLOWER
I20260621 07:47:17.007647 3008 raft_consensus.cc:374] T e22fb9a57eca4954848993c2abfdfa6a P 781c0beb96e8435a9dc711b0b70eb1da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:17.007647 3009 raft_consensus.cc:374] T e22fb9a57eca4954848993c2abfdfa6a P 1cd096c3570149679c0417ccd70dfe74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:17.007689 3008 raft_consensus.cc:729] T e22fb9a57eca4954848993c2abfdfa6a P 781c0beb96e8435a9dc711b0b70eb1da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 781c0beb96e8435a9dc711b0b70eb1da, State: Initialized, Role: FOLLOWER
I20260621 07:47:17.007689 3009 raft_consensus.cc:729] T e22fb9a57eca4954848993c2abfdfa6a P 1cd096c3570149679c0417ccd70dfe74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cd096c3570149679c0417ccd70dfe74, State: Initialized, Role: FOLLOWER
I20260621 07:47:17.007995 3010 consensus_queue.cc:260] T e22fb9a57eca4954848993c2abfdfa6a P 1750da561edc402883853661828d10c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } }
I20260621 07:47:17.008041 3008 consensus_queue.cc:260] T e22fb9a57eca4954848993c2abfdfa6a P 781c0beb96e8435a9dc711b0b70eb1da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } }
I20260621 07:47:17.008051 3009 consensus_queue.cc:260] T e22fb9a57eca4954848993c2abfdfa6a P 1cd096c3570149679c0417ccd70dfe74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } }
I20260621 07:47:17.008520 2705 heartbeater.cc:503] Master 127.0.192.254:40179 was elected leader, sending a full tablet report...
I20260621 07:47:17.008810 3010 ts_tablet_manager.cc:1428] T e22fb9a57eca4954848993c2abfdfa6a P 1750da561edc402883853661828d10c6: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260621 07:47:17.008826 3008 ts_tablet_manager.cc:1428] T e22fb9a57eca4954848993c2abfdfa6a P 781c0beb96e8435a9dc711b0b70eb1da: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260621 07:47:17.009104 2845 heartbeater.cc:503] Master 127.0.192.254:40179 was elected leader, sending a full tablet report...
I20260621 07:47:17.009521 2985 heartbeater.cc:503] Master 127.0.192.254:40179 was elected leader, sending a full tablet report...
I20260621 07:47:17.010679 3009 ts_tablet_manager.cc:1428] T e22fb9a57eca4954848993c2abfdfa6a P 1cd096c3570149679c0417ccd70dfe74: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260621 07:47:17.053630 3015 raft_consensus.cc:482] T e22fb9a57eca4954848993c2abfdfa6a P 1cd096c3570149679c0417ccd70dfe74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:47:17.053814 3015 raft_consensus.cc:504] T e22fb9a57eca4954848993c2abfdfa6a P 1cd096c3570149679c0417ccd70dfe74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } }
I20260621 07:47:17.054840 3015 leader_election.cc:288] T e22fb9a57eca4954848993c2abfdfa6a P 1cd096c3570149679c0417ccd70dfe74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 781c0beb96e8435a9dc711b0b70eb1da (127.0.192.194:40397), 1750da561edc402883853661828d10c6 (127.0.192.193:45443)
I20260621 07:47:17.058835 2878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:17.055054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:40397 (local address 127.0.192.195:46063)
0621 07:47:17.055183 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:17.055196 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:17.055271 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:17.055580 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:17.055584 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:17.055599 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:17.055713 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:17.055717 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:17.056457 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:17.056459 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:17.057440 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:17.057449 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:17.058435 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:17.058437 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:17.058524 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:17.058547 (+ 23us) client_negotiation.cc:769] Sending connection context
0621 07:47:17.058600 (+ 53us) client_negotiation.cc:240] Negotiation successful
0621 07:47:17.058694 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260621 07:47:17.058826 3004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:17.055163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:46063 (local address 127.0.192.194:40397)
0621 07:47:17.055270 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:17.055275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:17.055286 (+ 11us) server_negotiation.cc:407] Connection header received
0621 07:47:17.055424 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:17.055426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:17.055461 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:17.055557 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:17.055840 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:17.056346 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:17.057719 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:17.058315 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:17.058441 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:17.058489 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:17.058645 (+ 156us) server_negotiation.cc:299] Negotiation successful
0621 07:47:17.058711 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 07:47:17.059069 3017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:17.055054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:45443 (local address 127.0.192.195:54557)
0621 07:47:17.055438 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:17.055447 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:17.055511 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:17.055789 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:17.055794 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:17.055820 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:17.056245 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:17.056251 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:17.056987 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:17.056990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:17.057891 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:17.057901 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:17.058693 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:17.058696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:17.058775 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:17.058793 (+ 18us) client_negotiation.cc:769] Sending connection context
0621 07:47:17.058864 (+ 71us) client_negotiation.cc:240] Negotiation successful
0621 07:47:17.058922 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":96,"thread_start_us":89,"threads_started":1}
I20260621 07:47:17.059074 3006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:17.055241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:54557 (local address 127.0.192.193:45443)
0621 07:47:17.055375 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:17.055379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:17.055519 (+ 140us) server_negotiation.cc:407] Connection header received
0621 07:47:17.055636 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:17.055639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:17.055684 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:17.055742 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:17.056376 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:17.056882 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:17.057996 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:17.058559 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:17.058683 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:17.058722 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:17.058899 (+ 177us) server_negotiation.cc:299] Negotiation successful
0621 07:47:17.058972 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 07:47:17.059649 2799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e22fb9a57eca4954848993c2abfdfa6a" candidate_uuid: "1cd096c3570149679c0417ccd70dfe74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "781c0beb96e8435a9dc711b0b70eb1da" is_pre_election: true
I20260621 07:47:17.059641 2659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e22fb9a57eca4954848993c2abfdfa6a" candidate_uuid: "1cd096c3570149679c0417ccd70dfe74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1750da561edc402883853661828d10c6" is_pre_election: true
I20260621 07:47:17.059880 2659 raft_consensus.cc:2454] T e22fb9a57eca4954848993c2abfdfa6a P 1750da561edc402883853661828d10c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cd096c3570149679c0417ccd70dfe74 in term 0.
I20260621 07:47:17.059899 2799 raft_consensus.cc:2454] T e22fb9a57eca4954848993c2abfdfa6a P 781c0beb96e8435a9dc711b0b70eb1da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cd096c3570149679c0417ccd70dfe74 in term 0.
I20260621 07:47:17.060340 2871 leader_election.cc:302] T e22fb9a57eca4954848993c2abfdfa6a P 1cd096c3570149679c0417ccd70dfe74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1750da561edc402883853661828d10c6, 1cd096c3570149679c0417ccd70dfe74, 781c0beb96e8435a9dc711b0b70eb1da; no voters:
I20260621 07:47:17.060624 3015 raft_consensus.cc:2781] T e22fb9a57eca4954848993c2abfdfa6a P 1cd096c3570149679c0417ccd70dfe74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:47:17.060721 3015 raft_consensus.cc:482] T e22fb9a57eca4954848993c2abfdfa6a P 1cd096c3570149679c0417ccd70dfe74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:47:17.060774 3015 raft_consensus.cc:3037] T e22fb9a57eca4954848993c2abfdfa6a P 1cd096c3570149679c0417ccd70dfe74 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:17.061713 3015 raft_consensus.cc:504] T e22fb9a57eca4954848993c2abfdfa6a P 1cd096c3570149679c0417ccd70dfe74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } }
I20260621 07:47:17.062059 3015 leader_election.cc:288] T e22fb9a57eca4954848993c2abfdfa6a P 1cd096c3570149679c0417ccd70dfe74 [CANDIDATE]: Term 1 election: Requested vote from peers 781c0beb96e8435a9dc711b0b70eb1da (127.0.192.194:40397), 1750da561edc402883853661828d10c6 (127.0.192.193:45443)
I20260621 07:47:17.062335 2659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e22fb9a57eca4954848993c2abfdfa6a" candidate_uuid: "1cd096c3570149679c0417ccd70dfe74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1750da561edc402883853661828d10c6"
I20260621 07:47:17.062348 2799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e22fb9a57eca4954848993c2abfdfa6a" candidate_uuid: "1cd096c3570149679c0417ccd70dfe74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "781c0beb96e8435a9dc711b0b70eb1da"
I20260621 07:47:17.062459 2659 raft_consensus.cc:3037] T e22fb9a57eca4954848993c2abfdfa6a P 1750da561edc402883853661828d10c6 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:17.062474 2799 raft_consensus.cc:3037] T e22fb9a57eca4954848993c2abfdfa6a P 781c0beb96e8435a9dc711b0b70eb1da [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:17.063406 2659 raft_consensus.cc:2454] T e22fb9a57eca4954848993c2abfdfa6a P 1750da561edc402883853661828d10c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cd096c3570149679c0417ccd70dfe74 in term 1.
I20260621 07:47:17.063710 2871 leader_election.cc:302] T e22fb9a57eca4954848993c2abfdfa6a P 1cd096c3570149679c0417ccd70dfe74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1750da561edc402883853661828d10c6, 1cd096c3570149679c0417ccd70dfe74; no voters:
I20260621 07:47:17.063877 2799 raft_consensus.cc:2454] T e22fb9a57eca4954848993c2abfdfa6a P 781c0beb96e8435a9dc711b0b70eb1da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cd096c3570149679c0417ccd70dfe74 in term 1.
I20260621 07:47:17.063894 3015 raft_consensus.cc:2781] T e22fb9a57eca4954848993c2abfdfa6a P 1cd096c3570149679c0417ccd70dfe74 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:47:17.064108 3015 raft_consensus.cc:686] T e22fb9a57eca4954848993c2abfdfa6a P 1cd096c3570149679c0417ccd70dfe74 [term 1 LEADER]: Becoming Leader. State: Replica: 1cd096c3570149679c0417ccd70dfe74, State: Running, Role: LEADER
I20260621 07:47:17.064384 3015 consensus_queue.cc:237] T e22fb9a57eca4954848993c2abfdfa6a P 1cd096c3570149679c0417ccd70dfe74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } }
W20260621 07:47:17.066196 2706 tablet.cc:2369] T e22fb9a57eca4954848993c2abfdfa6a P 1750da561edc402883853661828d10c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:47:17.067160 2502 catalog_manager.cc:5495] T e22fb9a57eca4954848993c2abfdfa6a P 1cd096c3570149679c0417ccd70dfe74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cd096c3570149679c0417ccd70dfe74 (127.0.192.195). New cstate: current_term: 1 leader_uuid: "1cd096c3570149679c0417ccd70dfe74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } health_report { overall_health: HEALTHY } } }
Jun 21 07:47:17 dist-test-slave-lzx3 krb5kdc[2457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:47:17.108484 3025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:17.101761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:40179 (local address 127.0.0.1:58252)
0621 07:47:17.101996 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:17.102005 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:17.102072 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:17.102264 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:17.102266 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:17.102563 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:17.102638 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:17.102643 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:17.103311 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:17.103313 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:17.103837 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:17.103843 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:17.104220 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:17.104221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:17.104271 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:17.104849 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:17.104859 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:17.106038 (+ 1179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:17.107365 (+ 1327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:17.107372 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:17.107376 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:17.107577 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:17.107792 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:17.107794 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:17.107795 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:17.107856 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:17.108130 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:17.108134 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:17.108208 (+ 74us) client_negotiation.cc:769] Sending connection context
0621 07:47:17.108281 (+ 73us) client_negotiation.cc:240] Negotiation successful
0621 07:47:17.108364 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":84,"threads_started":1}
I20260621 07:47:17.108505 2551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:17.101897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58252 (local address 127.0.192.254:40179)
0621 07:47:17.102025 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:17.102028 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:17.102044 (+ 16us) server_negotiation.cc:407] Connection header received
0621 07:47:17.102142 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:17.102144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:17.102175 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:17.102233 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:17.102756 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:17.103223 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:17.103941 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:17.104138 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:17.104199 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:17.106143 (+ 1944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:17.106146 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:17.106147 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:17.106165 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:17.107276 (+ 1111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:17.107654 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:17.107656 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:17.107657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:17.107717 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:17.107926 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:17.107928 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:17.107929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:17.108055 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:17.108117 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:17.108329 (+ 212us) server_negotiation.cc:299] Negotiation successful
0621 07:47:17.108410 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 07:47:17.111574 2502 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:58252:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260621 07:47:17.112120 2502 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:47:17.116245 2779 tablet_service.cc:1467] Processing CreateTablet for tablet 7b1164f98d514ac1b0c4013f1292a717 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d173a88727374ec4ad1e48db0534e9a0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 07:47:17.116197 2639 tablet_service.cc:1467] Processing CreateTablet for tablet 7b1164f98d514ac1b0c4013f1292a717 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d173a88727374ec4ad1e48db0534e9a0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 07:47:17.116336 2919 tablet_service.cc:1467] Processing CreateTablet for tablet 7b1164f98d514ac1b0c4013f1292a717 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d173a88727374ec4ad1e48db0534e9a0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 07:47:17.116815 2779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b1164f98d514ac1b0c4013f1292a717. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:17.116815 2639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b1164f98d514ac1b0c4013f1292a717. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:17.116904 2919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b1164f98d514ac1b0c4013f1292a717. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:17.118983 3009 tablet_bootstrap.cc:492] T 7b1164f98d514ac1b0c4013f1292a717 P 1cd096c3570149679c0417ccd70dfe74: Bootstrap starting.
I20260621 07:47:17.119298 3010 tablet_bootstrap.cc:492] T 7b1164f98d514ac1b0c4013f1292a717 P 1750da561edc402883853661828d10c6: Bootstrap starting.
I20260621 07:47:17.119576 3008 tablet_bootstrap.cc:492] T 7b1164f98d514ac1b0c4013f1292a717 P 781c0beb96e8435a9dc711b0b70eb1da: Bootstrap starting.
I20260621 07:47:17.119879 3009 tablet_bootstrap.cc:654] T 7b1164f98d514ac1b0c4013f1292a717 P 1cd096c3570149679c0417ccd70dfe74: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:17.120149 3010 tablet_bootstrap.cc:654] T 7b1164f98d514ac1b0c4013f1292a717 P 1750da561edc402883853661828d10c6: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:17.120424 3008 tablet_bootstrap.cc:654] T 7b1164f98d514ac1b0c4013f1292a717 P 781c0beb96e8435a9dc711b0b70eb1da: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:17.121285 3009 tablet_bootstrap.cc:492] T 7b1164f98d514ac1b0c4013f1292a717 P 1cd096c3570149679c0417ccd70dfe74: No bootstrap required, opened a new log
I20260621 07:47:17.121312 3010 tablet_bootstrap.cc:492] T 7b1164f98d514ac1b0c4013f1292a717 P 1750da561edc402883853661828d10c6: No bootstrap required, opened a new log
I20260621 07:47:17.121408 3009 ts_tablet_manager.cc:1397] T 7b1164f98d514ac1b0c4013f1292a717 P 1cd096c3570149679c0417ccd70dfe74: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 07:47:17.121413 3010 ts_tablet_manager.cc:1397] T 7b1164f98d514ac1b0c4013f1292a717 P 1750da561edc402883853661828d10c6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:17.121675 3008 tablet_bootstrap.cc:492] T 7b1164f98d514ac1b0c4013f1292a717 P 781c0beb96e8435a9dc711b0b70eb1da: No bootstrap required, opened a new log
I20260621 07:47:17.121763 3008 ts_tablet_manager.cc:1397] T 7b1164f98d514ac1b0c4013f1292a717 P 781c0beb96e8435a9dc711b0b70eb1da: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 07:47:17.121886 3009 raft_consensus.cc:348] T 7b1164f98d514ac1b0c4013f1292a717 P 1cd096c3570149679c0417ccd70dfe74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } }
I20260621 07:47:17.122004 3009 raft_consensus.cc:374] T 7b1164f98d514ac1b0c4013f1292a717 P 1cd096c3570149679c0417ccd70dfe74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:17.122052 3009 raft_consensus.cc:729] T 7b1164f98d514ac1b0c4013f1292a717 P 1cd096c3570149679c0417ccd70dfe74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cd096c3570149679c0417ccd70dfe74, State: Initialized, Role: FOLLOWER
I20260621 07:47:17.122145 3010 raft_consensus.cc:348] T 7b1164f98d514ac1b0c4013f1292a717 P 1750da561edc402883853661828d10c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } }
I20260621 07:47:17.122160 3008 raft_consensus.cc:348] T 7b1164f98d514ac1b0c4013f1292a717 P 781c0beb96e8435a9dc711b0b70eb1da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } }
I20260621 07:47:17.122174 3009 consensus_queue.cc:260] T 7b1164f98d514ac1b0c4013f1292a717 P 1cd096c3570149679c0417ccd70dfe74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } }
I20260621 07:47:17.122251 3010 raft_consensus.cc:374] T 7b1164f98d514ac1b0c4013f1292a717 P 1750da561edc402883853661828d10c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:17.122251 3008 raft_consensus.cc:374] T 7b1164f98d514ac1b0c4013f1292a717 P 781c0beb96e8435a9dc711b0b70eb1da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:17.122283 3008 raft_consensus.cc:729] T 7b1164f98d514ac1b0c4013f1292a717 P 781c0beb96e8435a9dc711b0b70eb1da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 781c0beb96e8435a9dc711b0b70eb1da, State: Initialized, Role: FOLLOWER
I20260621 07:47:17.122283 3010 raft_consensus.cc:729] T 7b1164f98d514ac1b0c4013f1292a717 P 1750da561edc402883853661828d10c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1750da561edc402883853661828d10c6, State: Initialized, Role: FOLLOWER
I20260621 07:47:17.122388 3008 consensus_queue.cc:260] T 7b1164f98d514ac1b0c4013f1292a717 P 781c0beb96e8435a9dc711b0b70eb1da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } }
I20260621 07:47:17.122388 3010 consensus_queue.cc:260] T 7b1164f98d514ac1b0c4013f1292a717 P 1750da561edc402883853661828d10c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } }
I20260621 07:47:17.122632 3008 ts_tablet_manager.cc:1428] T 7b1164f98d514ac1b0c4013f1292a717 P 781c0beb96e8435a9dc711b0b70eb1da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260621 07:47:17.122611 3014 tablet_replica.cc:439] T 7b1164f98d514ac1b0c4013f1292a717 P 1750da561edc402883853661828d10c6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } } }
I20260621 07:47:17.122767 3014 tablet_replica.cc:442] T 7b1164f98d514ac1b0c4013f1292a717 P 1750da561edc402883853661828d10c6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:47:17.122865 3016 tablet_replica.cc:439] T 7b1164f98d514ac1b0c4013f1292a717 P 781c0beb96e8435a9dc711b0b70eb1da: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } } }
I20260621 07:47:17.122982 3016 tablet_replica.cc:442] T 7b1164f98d514ac1b0c4013f1292a717 P 781c0beb96e8435a9dc711b0b70eb1da: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:47:17.122619 3010 ts_tablet_manager.cc:1428] T 7b1164f98d514ac1b0c4013f1292a717 P 1750da561edc402883853661828d10c6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260621 07:47:17.122442 3009 ts_tablet_manager.cc:1428] T 7b1164f98d514ac1b0c4013f1292a717 P 1cd096c3570149679c0417ccd70dfe74: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260621 07:47:17.122934 3015 tablet_replica.cc:439] T 7b1164f98d514ac1b0c4013f1292a717 P 1cd096c3570149679c0417ccd70dfe74: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } } }
I20260621 07:47:17.123219 3015 tablet_replica.cc:442] T 7b1164f98d514ac1b0c4013f1292a717 P 1cd096c3570149679c0417ccd70dfe74: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:47:17.146497 3015 raft_consensus.cc:482] T 7b1164f98d514ac1b0c4013f1292a717 P 1cd096c3570149679c0417ccd70dfe74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:47:17.146579 3015 raft_consensus.cc:504] T 7b1164f98d514ac1b0c4013f1292a717 P 1cd096c3570149679c0417ccd70dfe74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } }
I20260621 07:47:17.146809 3015 leader_election.cc:288] T 7b1164f98d514ac1b0c4013f1292a717 P 1cd096c3570149679c0417ccd70dfe74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1750da561edc402883853661828d10c6 (127.0.192.193:45443), 781c0beb96e8435a9dc711b0b70eb1da (127.0.192.194:40397)
I20260621 07:47:17.147159 2659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1164f98d514ac1b0c4013f1292a717" candidate_uuid: "1cd096c3570149679c0417ccd70dfe74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1750da561edc402883853661828d10c6" is_pre_election: true
I20260621 07:47:17.147158 2799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1164f98d514ac1b0c4013f1292a717" candidate_uuid: "1cd096c3570149679c0417ccd70dfe74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "781c0beb96e8435a9dc711b0b70eb1da" is_pre_election: true
I20260621 07:47:17.147269 2659 raft_consensus.cc:2454] T 7b1164f98d514ac1b0c4013f1292a717 P 1750da561edc402883853661828d10c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cd096c3570149679c0417ccd70dfe74 in term 0.
I20260621 07:47:17.147269 2799 raft_consensus.cc:2454] T 7b1164f98d514ac1b0c4013f1292a717 P 781c0beb96e8435a9dc711b0b70eb1da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cd096c3570149679c0417ccd70dfe74 in term 0.
I20260621 07:47:17.147547 2870 leader_election.cc:302] T 7b1164f98d514ac1b0c4013f1292a717 P 1cd096c3570149679c0417ccd70dfe74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1750da561edc402883853661828d10c6, 1cd096c3570149679c0417ccd70dfe74, 781c0beb96e8435a9dc711b0b70eb1da; no voters:
I20260621 07:47:17.147686 3015 raft_consensus.cc:2781] T 7b1164f98d514ac1b0c4013f1292a717 P 1cd096c3570149679c0417ccd70dfe74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:47:17.147735 3015 raft_consensus.cc:482] T 7b1164f98d514ac1b0c4013f1292a717 P 1cd096c3570149679c0417ccd70dfe74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:47:17.147779 3015 raft_consensus.cc:3037] T 7b1164f98d514ac1b0c4013f1292a717 P 1cd096c3570149679c0417ccd70dfe74 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:17.148501 3015 raft_consensus.cc:504] T 7b1164f98d514ac1b0c4013f1292a717 P 1cd096c3570149679c0417ccd70dfe74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } }
I20260621 07:47:17.148859 3015 leader_election.cc:288] T 7b1164f98d514ac1b0c4013f1292a717 P 1cd096c3570149679c0417ccd70dfe74 [CANDIDATE]: Term 1 election: Requested vote from peers 1750da561edc402883853661828d10c6 (127.0.192.193:45443), 781c0beb96e8435a9dc711b0b70eb1da (127.0.192.194:40397)
I20260621 07:47:17.149118 2659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1164f98d514ac1b0c4013f1292a717" candidate_uuid: "1cd096c3570149679c0417ccd70dfe74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1750da561edc402883853661828d10c6"
I20260621 07:47:17.149140 2799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b1164f98d514ac1b0c4013f1292a717" candidate_uuid: "1cd096c3570149679c0417ccd70dfe74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "781c0beb96e8435a9dc711b0b70eb1da"
I20260621 07:47:17.149207 2659 raft_consensus.cc:3037] T 7b1164f98d514ac1b0c4013f1292a717 P 1750da561edc402883853661828d10c6 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:17.149262 2799 raft_consensus.cc:3037] T 7b1164f98d514ac1b0c4013f1292a717 P 781c0beb96e8435a9dc711b0b70eb1da [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:17.150243 2659 raft_consensus.cc:2454] T 7b1164f98d514ac1b0c4013f1292a717 P 1750da561edc402883853661828d10c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cd096c3570149679c0417ccd70dfe74 in term 1.
I20260621 07:47:17.150257 2799 raft_consensus.cc:2454] T 7b1164f98d514ac1b0c4013f1292a717 P 781c0beb96e8435a9dc711b0b70eb1da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cd096c3570149679c0417ccd70dfe74 in term 1.
I20260621 07:47:17.150475 2871 leader_election.cc:302] T 7b1164f98d514ac1b0c4013f1292a717 P 1cd096c3570149679c0417ccd70dfe74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1750da561edc402883853661828d10c6, 1cd096c3570149679c0417ccd70dfe74; no voters:
I20260621 07:47:17.150628 3015 raft_consensus.cc:2781] T 7b1164f98d514ac1b0c4013f1292a717 P 1cd096c3570149679c0417ccd70dfe74 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:47:17.150709 3015 raft_consensus.cc:686] T 7b1164f98d514ac1b0c4013f1292a717 P 1cd096c3570149679c0417ccd70dfe74 [term 1 LEADER]: Becoming Leader. State: Replica: 1cd096c3570149679c0417ccd70dfe74, State: Running, Role: LEADER
I20260621 07:47:17.150817 3015 consensus_queue.cc:237] T 7b1164f98d514ac1b0c4013f1292a717 P 1cd096c3570149679c0417ccd70dfe74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } }
I20260621 07:47:17.151340 3018 tablet_replica.cc:439] T 7b1164f98d514ac1b0c4013f1292a717 P 1cd096c3570149679c0417ccd70dfe74: TxnStatusTablet state changed. Reason: New leader 1cd096c3570149679c0417ccd70dfe74. Latest consensus state: current_term: 1 leader_uuid: "1cd096c3570149679c0417ccd70dfe74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } } }
I20260621 07:47:17.151443 3018 tablet_replica.cc:442] T 7b1164f98d514ac1b0c4013f1292a717 P 1cd096c3570149679c0417ccd70dfe74: This TxnStatusTablet replica's current role is: LEADER
I20260621 07:47:17.151688 3030 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260621 07:47:17.151875 3030 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260621 07:47:17.152191 2499 catalog_manager.cc:5495] T 7b1164f98d514ac1b0c4013f1292a717 P 1cd096c3570149679c0417ccd70dfe74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cd096c3570149679c0417ccd70dfe74 (127.0.192.195). New cstate: current_term: 1 leader_uuid: "1cd096c3570149679c0417ccd70dfe74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } health_report { overall_health: HEALTHY } } }
I20260621 07:47:17.166944 3025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:17.163441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.195:36063 (local address 127.0.0.1:51414)
0621 07:47:17.163607 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:17.163623 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:17.163734 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:17.164069 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:17.164072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:17.164078 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 07:47:17.164165 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:17.164172 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:17.164914 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:17.164920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:17.165540 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:17.165547 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:17.165982 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:17.165986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:17.166080 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:17.166121 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 07:47:17.166703 (+ 582us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 07:47:17.166716 (+ 13us) client_negotiation.cc:769] Sending connection context
0621 07:47:17.166777 (+ 61us) client_negotiation.cc:240] Negotiation successful
0621 07:47:17.166843 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260621 07:47:17.166940 3007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:17.163527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51414 (local address 127.0.192.195:36063)
0621 07:47:17.163620 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:17.163625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:17.163670 (+ 45us) server_negotiation.cc:407] Connection header received
0621 07:47:17.163867 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:17.163870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:17.163936 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:17.164059 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:47:17.164271 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:17.164818 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:17.165667 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:17.165874 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:17.165983 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:17.166262 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:47:17.166611 (+ 349us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:47:17.166724 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:17.166786 (+ 62us) server_negotiation.cc:299] Negotiation successful
0621 07:47:17.166830 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260621 07:47:17.171416 2799 raft_consensus.cc:1261] T 7b1164f98d514ac1b0c4013f1292a717 P 781c0beb96e8435a9dc711b0b70eb1da [term 1 FOLLOWER]: Refusing update from remote peer 1cd096c3570149679c0417ccd70dfe74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:47:17.171432 2659 raft_consensus.cc:1261] T 7b1164f98d514ac1b0c4013f1292a717 P 1750da561edc402883853661828d10c6 [term 1 FOLLOWER]: Refusing update from remote peer 1cd096c3570149679c0417ccd70dfe74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:47:17.171967 3018 consensus_queue.cc:1035] T 7b1164f98d514ac1b0c4013f1292a717 P 1cd096c3570149679c0417ccd70dfe74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:47:17.172184 3015 consensus_queue.cc:1035] T 7b1164f98d514ac1b0c4013f1292a717 P 1cd096c3570149679c0417ccd70dfe74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:47:17.175297 3016 tablet_replica.cc:439] T 7b1164f98d514ac1b0c4013f1292a717 P 781c0beb96e8435a9dc711b0b70eb1da: TxnStatusTablet state changed. Reason: New leader 1cd096c3570149679c0417ccd70dfe74. Latest consensus state: current_term: 1 leader_uuid: "1cd096c3570149679c0417ccd70dfe74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } } }
I20260621 07:47:17.175402 3016 tablet_replica.cc:442] T 7b1164f98d514ac1b0c4013f1292a717 P 781c0beb96e8435a9dc711b0b70eb1da: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:47:17.175434 3014 tablet_replica.cc:439] T 7b1164f98d514ac1b0c4013f1292a717 P 1750da561edc402883853661828d10c6: TxnStatusTablet state changed. Reason: New leader 1cd096c3570149679c0417ccd70dfe74. Latest consensus state: current_term: 1 leader_uuid: "1cd096c3570149679c0417ccd70dfe74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } } }
I20260621 07:47:17.175524 3014 tablet_replica.cc:442] T 7b1164f98d514ac1b0c4013f1292a717 P 1750da561edc402883853661828d10c6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:47:17.178771 3015 tablet_replica.cc:439] T 7b1164f98d514ac1b0c4013f1292a717 P 1cd096c3570149679c0417ccd70dfe74: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1cd096c3570149679c0417ccd70dfe74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } } }
I20260621 07:47:17.178859 3015 tablet_replica.cc:442] T 7b1164f98d514ac1b0c4013f1292a717 P 1cd096c3570149679c0417ccd70dfe74: This TxnStatusTablet replica's current role is: LEADER
I20260621 07:47:17.179157 3015 tablet_replica.cc:439] T 7b1164f98d514ac1b0c4013f1292a717 P 1cd096c3570149679c0417ccd70dfe74: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1cd096c3570149679c0417ccd70dfe74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } } }
I20260621 07:47:17.179239 3015 tablet_replica.cc:442] T 7b1164f98d514ac1b0c4013f1292a717 P 1cd096c3570149679c0417ccd70dfe74: This TxnStatusTablet replica's current role is: LEADER
I20260621 07:47:17.179366 3033 mvcc.cc:205] Tried to move back new op lower bound from 7299186840248328192 to 7299186840170672128. Current Snapshot: MvccSnapshot[applied={T|T < 7299186840248328192}]
I20260621 07:47:17.179423 3016 tablet_replica.cc:439] T 7b1164f98d514ac1b0c4013f1292a717 P 781c0beb96e8435a9dc711b0b70eb1da: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1cd096c3570149679c0417ccd70dfe74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } } }
I20260621 07:47:17.179536 3016 tablet_replica.cc:442] T 7b1164f98d514ac1b0c4013f1292a717 P 781c0beb96e8435a9dc711b0b70eb1da: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:47:17.180091 3014 tablet_replica.cc:439] T 7b1164f98d514ac1b0c4013f1292a717 P 1750da561edc402883853661828d10c6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1cd096c3570149679c0417ccd70dfe74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } } }
I20260621 07:47:17.180174 3014 tablet_replica.cc:442] T 7b1164f98d514ac1b0c4013f1292a717 P 1750da561edc402883853661828d10c6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:47:17.180271 3035 mvcc.cc:205] Tried to move back new op lower bound from 7299186840248328192 to 7299186840170672128. Current Snapshot: MvccSnapshot[applied={T|T < 7299186840248328192}]
I20260621 07:47:17.182328 3034 mvcc.cc:205] Tried to move back new op lower bound from 7299186840248328192 to 7299186840170672128. Current Snapshot: MvccSnapshot[applied={T|T < 7299186840248328192}]
W20260621 07:47:17.188482 2986 tablet.cc:2369] T e22fb9a57eca4954848993c2abfdfa6a P 1cd096c3570149679c0417ccd70dfe74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:47:17.190403 3007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:17.187165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51416 (local address 127.0.192.195:36063)
0621 07:47:17.187281 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:17.187284 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:17.187316 (+ 32us) server_negotiation.cc:407] Connection header received
0621 07:47:17.187442 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:17.187445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:17.187496 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:17.187565 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:47:17.187774 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:17.188455 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:17.189568 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:17.189737 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:17.189804 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:17.190066 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:47:17.190114 (+ 48us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:47:17.190158 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:17.190275 (+ 117us) server_negotiation.cc:299] Negotiation successful
0621 07:47:17.190305 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 07:47:17.199304 3007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:17.196358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:49955 (local address 127.0.192.195:36063)
0621 07:47:17.196444 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:17.196448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:17.196615 (+ 167us) server_negotiation.cc:407] Connection header received
0621 07:47:17.196715 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:17.196732 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:17.196768 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:17.196845 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:47:17.197046 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:17.197585 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:17.198406 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:17.198611 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:17.198706 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:17.198946 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:47:17.199002 (+ 56us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:47:17.199077 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:17.199147 (+ 70us) server_negotiation.cc:299] Negotiation successful
0621 07:47:17.199185 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":36}
I20260621 07:47:17.199304 2991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:17.196309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.195:36063 (local address 127.0.192.195:49955)
0621 07:47:17.196537 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:17.196549 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:17.196650 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:17.196850 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:17.196852 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:17.196857 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 07:47:17.196945 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:17.196950 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:17.197661 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:17.197666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:17.198302 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:17.198309 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:17.198702 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:17.198705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:17.198783 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:17.198822 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 07:47:17.199079 (+ 257us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 07:47:17.199085 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 07:47:17.199141 (+ 56us) client_negotiation.cc:240] Negotiation successful
0621 07:47:17.199202 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":4}
I20260621 07:47:17.204785 2659 raft_consensus.cc:1261] T e22fb9a57eca4954848993c2abfdfa6a P 1750da561edc402883853661828d10c6 [term 1 FOLLOWER]: Refusing update from remote peer 1cd096c3570149679c0417ccd70dfe74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:47:17.204794 2799 raft_consensus.cc:1261] T e22fb9a57eca4954848993c2abfdfa6a P 781c0beb96e8435a9dc711b0b70eb1da [term 1 FOLLOWER]: Refusing update from remote peer 1cd096c3570149679c0417ccd70dfe74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:47:17.205158 3018 consensus_queue.cc:1035] T e22fb9a57eca4954848993c2abfdfa6a P 1cd096c3570149679c0417ccd70dfe74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:47:17.205323 3015 consensus_queue.cc:1035] T e22fb9a57eca4954848993c2abfdfa6a P 1cd096c3570149679c0417ccd70dfe74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:47:17.241257 2501 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:47:17.241403 2501 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:47:17.243330 2501 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 147c497bf1a945a1a37de938664b2f47: Sending DeleteTablet for 3 replicas of tablet e22fb9a57eca4954848993c2abfdfa6a
I20260621 07:47:17.243909 2779 tablet_service.cc:1514] Processing DeleteTablet for tablet e22fb9a57eca4954848993c2abfdfa6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:47:17 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:58982
I20260621 07:47:17.244024 2918 tablet_service.cc:1514] Processing DeleteTablet for tablet e22fb9a57eca4954848993c2abfdfa6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:47:17 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:51408
I20260621 07:47:17.244287 2639 tablet_service.cc:1514] Processing DeleteTablet for tablet e22fb9a57eca4954848993c2abfdfa6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:47:17 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:58144
I20260621 07:47:17.244398 3049 tablet_replica.cc:330] T e22fb9a57eca4954848993c2abfdfa6a P 1cd096c3570149679c0417ccd70dfe74: stopping tablet replica
I20260621 07:47:17.244652 3049 raft_consensus.cc:2229] T e22fb9a57eca4954848993c2abfdfa6a P 1cd096c3570149679c0417ccd70dfe74 [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:47:17.244690 3050 tablet_replica.cc:330] T e22fb9a57eca4954848993c2abfdfa6a P 1750da561edc402883853661828d10c6: stopping tablet replica
I20260621 07:47:17.244940 3048 tablet_replica.cc:330] T e22fb9a57eca4954848993c2abfdfa6a P 781c0beb96e8435a9dc711b0b70eb1da: stopping tablet replica
I20260621 07:47:17.245044 3050 raft_consensus.cc:2229] T e22fb9a57eca4954848993c2abfdfa6a P 1750da561edc402883853661828d10c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:47:17.245163 3048 raft_consensus.cc:2229] T e22fb9a57eca4954848993c2abfdfa6a P 781c0beb96e8435a9dc711b0b70eb1da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:47:17.245117 2501 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58242:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260621 07:47:17.245314 3048 raft_consensus.cc:2258] T e22fb9a57eca4954848993c2abfdfa6a P 781c0beb96e8435a9dc711b0b70eb1da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:47:17.245404 3050 raft_consensus.cc:2258] T e22fb9a57eca4954848993c2abfdfa6a P 1750da561edc402883853661828d10c6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260621 07:47:17.245440 2501 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:47:17.246503 3050 ts_tablet_manager.cc:1905] T e22fb9a57eca4954848993c2abfdfa6a P 1750da561edc402883853661828d10c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:47:17.246507 3048 ts_tablet_manager.cc:1905] T e22fb9a57eca4954848993c2abfdfa6a P 781c0beb96e8435a9dc711b0b70eb1da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:47:17.246767 3049 raft_consensus.cc:2258] T e22fb9a57eca4954848993c2abfdfa6a P 1cd096c3570149679c0417ccd70dfe74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:47:17.249362 3050 ts_tablet_manager.cc:1918] T e22fb9a57eca4954848993c2abfdfa6a P 1750da561edc402883853661828d10c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 07:47:17.249495 3050 log.cc:1196] T e22fb9a57eca4954848993c2abfdfa6a P 1750da561edc402883853661828d10c6: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-0/wal/wals/e22fb9a57eca4954848993c2abfdfa6a
I20260621 07:47:17.249831 3050 ts_tablet_manager.cc:1939] T e22fb9a57eca4954848993c2abfdfa6a P 1750da561edc402883853661828d10c6: Deleting consensus metadata
I20260621 07:47:17.249832 3048 ts_tablet_manager.cc:1918] T e22fb9a57eca4954848993c2abfdfa6a P 781c0beb96e8435a9dc711b0b70eb1da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 07:47:17.249912 3048 log.cc:1196] T e22fb9a57eca4954848993c2abfdfa6a P 781c0beb96e8435a9dc711b0b70eb1da: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-1/wal/wals/e22fb9a57eca4954848993c2abfdfa6a
I20260621 07:47:17.250180 3048 ts_tablet_manager.cc:1939] T e22fb9a57eca4954848993c2abfdfa6a P 781c0beb96e8435a9dc711b0b70eb1da: Deleting consensus metadata
I20260621 07:47:17.250586 2779 tablet_service.cc:1467] Processing CreateTablet for tablet 5e145751d0c145529dbf0449ecf5c8d4 (DEFAULT_TABLE table=test-table [id=c3dc7c5fc356417da3632c84074ced3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:47:17.250814 2779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e145751d0c145529dbf0449ecf5c8d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:17.252277 3008 tablet_bootstrap.cc:492] T 5e145751d0c145529dbf0449ecf5c8d4 P 781c0beb96e8435a9dc711b0b70eb1da: Bootstrap starting.
I20260621 07:47:17.252427 2486 catalog_manager.cc:4841] TS 781c0beb96e8435a9dc711b0b70eb1da (127.0.192.194:40397): tablet e22fb9a57eca4954848993c2abfdfa6a (table test-table [id=d69123da9e894953b123c26f58c206cd]) successfully deleted
I20260621 07:47:17.252952 2487 catalog_manager.cc:4841] TS 1750da561edc402883853661828d10c6 (127.0.192.193:45443): tablet e22fb9a57eca4954848993c2abfdfa6a (table test-table [id=d69123da9e894953b123c26f58c206cd]) successfully deleted
I20260621 07:47:17.253206 3008 tablet_bootstrap.cc:654] T 5e145751d0c145529dbf0449ecf5c8d4 P 781c0beb96e8435a9dc711b0b70eb1da: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:17.253180 2918 tablet_service.cc:1467] Processing CreateTablet for tablet 5e145751d0c145529dbf0449ecf5c8d4 (DEFAULT_TABLE table=test-table [id=c3dc7c5fc356417da3632c84074ced3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:47:17.253410 2918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e145751d0c145529dbf0449ecf5c8d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:17.253863 2639 tablet_service.cc:1467] Processing CreateTablet for tablet 5e145751d0c145529dbf0449ecf5c8d4 (DEFAULT_TABLE table=test-table [id=c3dc7c5fc356417da3632c84074ced3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:47:17.254058 2639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e145751d0c145529dbf0449ecf5c8d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:17.254295 3049 ts_tablet_manager.cc:1905] T e22fb9a57eca4954848993c2abfdfa6a P 1cd096c3570149679c0417ccd70dfe74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:47:17.255637 3009 tablet_bootstrap.cc:492] T 5e145751d0c145529dbf0449ecf5c8d4 P 1cd096c3570149679c0417ccd70dfe74: Bootstrap starting.
I20260621 07:47:17.256032 3010 tablet_bootstrap.cc:492] T 5e145751d0c145529dbf0449ecf5c8d4 P 1750da561edc402883853661828d10c6: Bootstrap starting.
I20260621 07:47:17.256320 3009 tablet_bootstrap.cc:654] T 5e145751d0c145529dbf0449ecf5c8d4 P 1cd096c3570149679c0417ccd70dfe74: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:17.256671 3049 ts_tablet_manager.cc:1918] T e22fb9a57eca4954848993c2abfdfa6a P 1cd096c3570149679c0417ccd70dfe74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 07:47:17.256747 3049 log.cc:1196] T e22fb9a57eca4954848993c2abfdfa6a P 1cd096c3570149679c0417ccd70dfe74: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-2/wal/wals/e22fb9a57eca4954848993c2abfdfa6a
I20260621 07:47:17.256783 3008 tablet_bootstrap.cc:492] T 5e145751d0c145529dbf0449ecf5c8d4 P 781c0beb96e8435a9dc711b0b70eb1da: No bootstrap required, opened a new log
I20260621 07:47:17.256857 3008 ts_tablet_manager.cc:1397] T 5e145751d0c145529dbf0449ecf5c8d4 P 781c0beb96e8435a9dc711b0b70eb1da: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260621 07:47:17.256937 3010 tablet_bootstrap.cc:654] T 5e145751d0c145529dbf0449ecf5c8d4 P 1750da561edc402883853661828d10c6: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:17.257016 3049 ts_tablet_manager.cc:1939] T e22fb9a57eca4954848993c2abfdfa6a P 1cd096c3570149679c0417ccd70dfe74: Deleting consensus metadata
I20260621 07:47:17.257238 3008 raft_consensus.cc:348] T 5e145751d0c145529dbf0449ecf5c8d4 P 781c0beb96e8435a9dc711b0b70eb1da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } }
I20260621 07:47:17.257328 3008 raft_consensus.cc:374] T 5e145751d0c145529dbf0449ecf5c8d4 P 781c0beb96e8435a9dc711b0b70eb1da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:17.257359 3008 raft_consensus.cc:729] T 5e145751d0c145529dbf0449ecf5c8d4 P 781c0beb96e8435a9dc711b0b70eb1da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 781c0beb96e8435a9dc711b0b70eb1da, State: Initialized, Role: FOLLOWER
I20260621 07:47:17.257453 3008 consensus_queue.cc:260] T 5e145751d0c145529dbf0449ecf5c8d4 P 781c0beb96e8435a9dc711b0b70eb1da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } }
I20260621 07:47:17.257663 3008 ts_tablet_manager.cc:1428] T 5e145751d0c145529dbf0449ecf5c8d4 P 781c0beb96e8435a9dc711b0b70eb1da: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260621 07:47:17.257844 2487 catalog_manager.cc:4841] TS 1cd096c3570149679c0417ccd70dfe74 (127.0.192.195:36063): tablet e22fb9a57eca4954848993c2abfdfa6a (table test-table [id=d69123da9e894953b123c26f58c206cd]) successfully deleted
I20260621 07:47:17.261183 3010 tablet_bootstrap.cc:492] T 5e145751d0c145529dbf0449ecf5c8d4 P 1750da561edc402883853661828d10c6: No bootstrap required, opened a new log
I20260621 07:47:17.261286 3010 ts_tablet_manager.cc:1397] T 5e145751d0c145529dbf0449ecf5c8d4 P 1750da561edc402883853661828d10c6: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260621 07:47:17.261512 3009 tablet_bootstrap.cc:492] T 5e145751d0c145529dbf0449ecf5c8d4 P 1cd096c3570149679c0417ccd70dfe74: No bootstrap required, opened a new log
I20260621 07:47:17.261593 3009 ts_tablet_manager.cc:1397] T 5e145751d0c145529dbf0449ecf5c8d4 P 1cd096c3570149679c0417ccd70dfe74: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260621 07:47:17.261727 3010 raft_consensus.cc:348] T 5e145751d0c145529dbf0449ecf5c8d4 P 1750da561edc402883853661828d10c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } }
I20260621 07:47:17.261837 3010 raft_consensus.cc:374] T 5e145751d0c145529dbf0449ecf5c8d4 P 1750da561edc402883853661828d10c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:17.261871 3010 raft_consensus.cc:729] T 5e145751d0c145529dbf0449ecf5c8d4 P 1750da561edc402883853661828d10c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1750da561edc402883853661828d10c6, State: Initialized, Role: FOLLOWER
I20260621 07:47:17.261979 3010 consensus_queue.cc:260] T 5e145751d0c145529dbf0449ecf5c8d4 P 1750da561edc402883853661828d10c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } }
I20260621 07:47:17.261965 3009 raft_consensus.cc:348] T 5e145751d0c145529dbf0449ecf5c8d4 P 1cd096c3570149679c0417ccd70dfe74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } }
I20260621 07:47:17.262058 3009 raft_consensus.cc:374] T 5e145751d0c145529dbf0449ecf5c8d4 P 1cd096c3570149679c0417ccd70dfe74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:17.262086 3009 raft_consensus.cc:729] T 5e145751d0c145529dbf0449ecf5c8d4 P 1cd096c3570149679c0417ccd70dfe74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cd096c3570149679c0417ccd70dfe74, State: Initialized, Role: FOLLOWER
I20260621 07:47:17.262220 3010 ts_tablet_manager.cc:1428] T 5e145751d0c145529dbf0449ecf5c8d4 P 1750da561edc402883853661828d10c6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260621 07:47:17.262193 3009 consensus_queue.cc:260] T 5e145751d0c145529dbf0449ecf5c8d4 P 1cd096c3570149679c0417ccd70dfe74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } }
I20260621 07:47:17.262416 3009 ts_tablet_manager.cc:1428] T 5e145751d0c145529dbf0449ecf5c8d4 P 1cd096c3570149679c0417ccd70dfe74: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260621 07:47:17.331331 3016 raft_consensus.cc:482] T 5e145751d0c145529dbf0449ecf5c8d4 P 781c0beb96e8435a9dc711b0b70eb1da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:47:17.331467 3016 raft_consensus.cc:504] T 5e145751d0c145529dbf0449ecf5c8d4 P 781c0beb96e8435a9dc711b0b70eb1da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } }
I20260621 07:47:17.332553 3016 leader_election.cc:288] T 5e145751d0c145529dbf0449ecf5c8d4 P 781c0beb96e8435a9dc711b0b70eb1da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1750da561edc402883853661828d10c6 (127.0.192.193:45443), 1cd096c3570149679c0417ccd70dfe74 (127.0.192.195:36063)
I20260621 07:47:17.337066 3006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:17.332866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:33629 (local address 127.0.192.193:45443)
0621 07:47:17.333003 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:17.333008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:17.333103 (+ 95us) server_negotiation.cc:407] Connection header received
0621 07:47:17.333266 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:17.333270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:17.333324 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:17.333391 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:17.333820 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:17.334410 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:17.335905 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:17.336431 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:17.336494 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:17.336589 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:17.336873 (+ 284us) server_negotiation.cc:299] Negotiation successful
0621 07:47:17.336934 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 07:47:17.337082 3054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:17.332751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:45443 (local address 127.0.192.194:33629)
0621 07:47:17.333002 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:17.333018 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:17.333123 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:17.333547 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:17.333551 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:17.333569 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:17.333675 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:17.333680 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:17.334539 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:17.334541 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:17.335747 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:17.335758 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:17.336675 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:17.336677 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:17.336740 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:17.336788 (+ 48us) client_negotiation.cc:769] Sending connection context
0621 07:47:17.336857 (+ 69us) client_negotiation.cc:240] Negotiation successful
0621 07:47:17.336922 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":107,"threads_started":1}
I20260621 07:47:17.337486 2659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e145751d0c145529dbf0449ecf5c8d4" candidate_uuid: "781c0beb96e8435a9dc711b0b70eb1da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1750da561edc402883853661828d10c6" is_pre_election: true
I20260621 07:47:17.337646 2659 raft_consensus.cc:2454] T 5e145751d0c145529dbf0449ecf5c8d4 P 1750da561edc402883853661828d10c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 781c0beb96e8435a9dc711b0b70eb1da in term 0.
I20260621 07:47:17.337894 3055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:17.333132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.195:36063 (local address 127.0.192.194:35427)
0621 07:47:17.333409 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:17.333422 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:17.333511 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:17.333775 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:17.333779 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:17.333802 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:17.333897 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:17.333902 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:17.335061 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:17.335064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:17.336224 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:17.336232 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:17.337492 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:17.337498 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:17.337585 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:17.337605 (+ 20us) client_negotiation.cc:769] Sending connection context
0621 07:47:17.337711 (+ 106us) client_negotiation.cc:240] Negotiation successful
0621 07:47:17.337769 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":61,"threads_started":1}
I20260621 07:47:17.337935 3007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:17.333269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:35427 (local address 127.0.192.195:36063)
0621 07:47:17.333373 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:17.333377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:17.333493 (+ 116us) server_negotiation.cc:407] Connection header received
0621 07:47:17.333617 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:17.333620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:17.333667 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:17.333741 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:17.334000 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:17.334710 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:17.336350 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:17.336959 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:17.337087 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:17.337143 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:17.337732 (+ 589us) server_negotiation.cc:299] Negotiation successful
0621 07:47:17.337816 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260621 07:47:17.337993 2731 leader_election.cc:302] T 5e145751d0c145529dbf0449ecf5c8d4 P 781c0beb96e8435a9dc711b0b70eb1da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1750da561edc402883853661828d10c6, 781c0beb96e8435a9dc711b0b70eb1da; no voters:
I20260621 07:47:17.338274 3016 raft_consensus.cc:2781] T 5e145751d0c145529dbf0449ecf5c8d4 P 781c0beb96e8435a9dc711b0b70eb1da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:47:17.338322 3016 raft_consensus.cc:482] T 5e145751d0c145529dbf0449ecf5c8d4 P 781c0beb96e8435a9dc711b0b70eb1da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:47:17.338362 3016 raft_consensus.cc:3037] T 5e145751d0c145529dbf0449ecf5c8d4 P 781c0beb96e8435a9dc711b0b70eb1da [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:17.338512 2939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e145751d0c145529dbf0449ecf5c8d4" candidate_uuid: "781c0beb96e8435a9dc711b0b70eb1da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cd096c3570149679c0417ccd70dfe74" is_pre_election: true
I20260621 07:47:17.338711 2939 raft_consensus.cc:2454] T 5e145751d0c145529dbf0449ecf5c8d4 P 1cd096c3570149679c0417ccd70dfe74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 781c0beb96e8435a9dc711b0b70eb1da in term 0.
I20260621 07:47:17.339488 3016 raft_consensus.cc:504] T 5e145751d0c145529dbf0449ecf5c8d4 P 781c0beb96e8435a9dc711b0b70eb1da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } }
I20260621 07:47:17.339824 3016 leader_election.cc:288] T 5e145751d0c145529dbf0449ecf5c8d4 P 781c0beb96e8435a9dc711b0b70eb1da [CANDIDATE]: Term 1 election: Requested vote from peers 1750da561edc402883853661828d10c6 (127.0.192.193:45443), 1cd096c3570149679c0417ccd70dfe74 (127.0.192.195:36063)
I20260621 07:47:17.340143 2939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e145751d0c145529dbf0449ecf5c8d4" candidate_uuid: "781c0beb96e8435a9dc711b0b70eb1da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cd096c3570149679c0417ccd70dfe74"
I20260621 07:47:17.340183 2659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e145751d0c145529dbf0449ecf5c8d4" candidate_uuid: "781c0beb96e8435a9dc711b0b70eb1da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1750da561edc402883853661828d10c6"
I20260621 07:47:17.340250 2939 raft_consensus.cc:3037] T 5e145751d0c145529dbf0449ecf5c8d4 P 1cd096c3570149679c0417ccd70dfe74 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:17.340296 2659 raft_consensus.cc:3037] T 5e145751d0c145529dbf0449ecf5c8d4 P 1750da561edc402883853661828d10c6 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:17.341274 2659 raft_consensus.cc:2454] T 5e145751d0c145529dbf0449ecf5c8d4 P 1750da561edc402883853661828d10c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 781c0beb96e8435a9dc711b0b70eb1da in term 1.
I20260621 07:47:17.341346 2939 raft_consensus.cc:2454] T 5e145751d0c145529dbf0449ecf5c8d4 P 1cd096c3570149679c0417ccd70dfe74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 781c0beb96e8435a9dc711b0b70eb1da in term 1.
I20260621 07:47:17.341598 2731 leader_election.cc:302] T 5e145751d0c145529dbf0449ecf5c8d4 P 781c0beb96e8435a9dc711b0b70eb1da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cd096c3570149679c0417ccd70dfe74, 781c0beb96e8435a9dc711b0b70eb1da; no voters:
I20260621 07:47:17.341768 3016 raft_consensus.cc:2781] T 5e145751d0c145529dbf0449ecf5c8d4 P 781c0beb96e8435a9dc711b0b70eb1da [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:47:17.341926 3016 raft_consensus.cc:686] T 5e145751d0c145529dbf0449ecf5c8d4 P 781c0beb96e8435a9dc711b0b70eb1da [term 1 LEADER]: Becoming Leader. State: Replica: 781c0beb96e8435a9dc711b0b70eb1da, State: Running, Role: LEADER
I20260621 07:47:17.342136 3016 consensus_queue.cc:237] T 5e145751d0c145529dbf0449ecf5c8d4 P 781c0beb96e8435a9dc711b0b70eb1da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } }
I20260621 07:47:17.344627 2496 catalog_manager.cc:5495] T 5e145751d0c145529dbf0449ecf5c8d4 P 781c0beb96e8435a9dc711b0b70eb1da reported cstate change: term changed from 0 to 1, leader changed from <none> to 781c0beb96e8435a9dc711b0b70eb1da (127.0.192.194). New cstate: current_term: 1 leader_uuid: "781c0beb96e8435a9dc711b0b70eb1da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } health_report { overall_health: UNKNOWN } } }
I20260621 07:47:17.365991 3004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:17.362326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58984 (local address 127.0.192.194:40397)
0621 07:47:17.362426 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:17.362430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:17.362470 (+ 40us) server_negotiation.cc:407] Connection header received
0621 07:47:17.362595 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:17.362598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:17.362650 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:17.362786 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:47:17.362992 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:17.363736 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:17.364482 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:17.364794 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:17.364883 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:17.365115 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:47:17.365634 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:47:17.365748 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:17.365787 (+ 39us) server_negotiation.cc:299] Negotiation successful
0621 07:47:17.365836 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260621 07:47:17.375321 3007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:17.372025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:59117 (local address 127.0.192.195:36063)
0621 07:47:17.372189 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:17.372194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:17.372319 (+ 125us) server_negotiation.cc:407] Connection header received
0621 07:47:17.372402 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:17.372405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:17.372451 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:17.372529 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:47:17.372807 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:17.373542 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:17.374362 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:17.374660 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:17.374750 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:17.374968 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:47:17.375036 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:47:17.375101 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:17.375169 (+ 68us) server_negotiation.cc:299] Negotiation successful
0621 07:47:17.375209 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260621 07:47:17.375720 3060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:17.371891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.195:36063 (local address 127.0.192.194:59117)
0621 07:47:17.372237 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:17.372249 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:17.372326 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:17.372546 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:17.372548 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:17.372600 (+ 52us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 07:47:17.372700 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:17.372705 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:17.373649 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:17.373652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:17.374258 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:17.374265 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:17.374757 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:17.374759 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:17.374819 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:17.374854 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 07:47:17.375113 (+ 259us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 07:47:17.375117 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 07:47:17.375160 (+ 43us) client_negotiation.cc:240] Negotiation successful
0621 07:47:17.375211 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":94,"threads_started":1}
I20260621 07:47:17.381136 2659 raft_consensus.cc:1261] T 5e145751d0c145529dbf0449ecf5c8d4 P 1750da561edc402883853661828d10c6 [term 1 FOLLOWER]: Refusing update from remote peer 781c0beb96e8435a9dc711b0b70eb1da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:47:17.381368 2939 raft_consensus.cc:1261] T 5e145751d0c145529dbf0449ecf5c8d4 P 1cd096c3570149679c0417ccd70dfe74 [term 1 FOLLOWER]: Refusing update from remote peer 781c0beb96e8435a9dc711b0b70eb1da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:47:17.381551 3056 consensus_queue.cc:1035] T 5e145751d0c145529dbf0449ecf5c8d4 P 781c0beb96e8435a9dc711b0b70eb1da [LEADER]: Connected to new peer: Peer: permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:47:17.381819 3016 consensus_queue.cc:1035] T 5e145751d0c145529dbf0449ecf5c8d4 P 781c0beb96e8435a9dc711b0b70eb1da [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:47:17.401495 3004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:17.397915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:60625 (local address 127.0.192.194:40397)
0621 07:47:17.398144 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:17.398149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:17.398165 (+ 16us) server_negotiation.cc:407] Connection header received
0621 07:47:17.398209 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:17.398212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:17.398263 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:17.398354 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:47:17.398612 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:17.399303 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:17.400430 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:17.400742 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:17.400859 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:17.401047 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:47:17.401145 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:47:17.401213 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:17.401331 (+ 118us) server_negotiation.cc:299] Negotiation successful
0621 07:47:17.401380 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260621 07:47:17.401510 2991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:17.397732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:40397 (local address 127.0.192.195:60625)
0621 07:47:17.397843 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:17.397869 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:17.397947 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:17.398384 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:17.398386 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:17.398393 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 07:47:17.398468 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:17.398472 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:17.399433 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:17.399437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:17.400288 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:17.400298 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:17.400847 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:17.400851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:17.400910 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:17.400933 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 07:47:17.401225 (+ 292us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 07:47:17.401232 (+ 7us) client_negotiation.cc:769] Sending connection context
0621 07:47:17.401307 (+ 75us) client_negotiation.cc:240] Negotiation successful
0621 07:47:17.401382 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
W20260621 07:47:17.402477 3035 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260621 07:47:18.961335 2496 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:47:18.961509 2496 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:47:18.963673 2496 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 147c497bf1a945a1a37de938664b2f47: Sending DeleteTablet for 3 replicas of tablet 5e145751d0c145529dbf0449ecf5c8d4
I20260621 07:47:18.964145 2639 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e145751d0c145529dbf0449ecf5c8d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:47:18 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:58144
I20260621 07:47:18.964249 2779 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e145751d0c145529dbf0449ecf5c8d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:47:18 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:58982
I20260621 07:47:18.964283 2918 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e145751d0c145529dbf0449ecf5c8d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:47:18 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:51408
I20260621 07:47:18.964504 3086 tablet_replica.cc:330] T 5e145751d0c145529dbf0449ecf5c8d4 P 781c0beb96e8435a9dc711b0b70eb1da: stopping tablet replica
I20260621 07:47:18.964643 3086 raft_consensus.cc:2229] T 5e145751d0c145529dbf0449ecf5c8d4 P 781c0beb96e8435a9dc711b0b70eb1da [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:47:18.964900 3086 raft_consensus.cc:2258] T 5e145751d0c145529dbf0449ecf5c8d4 P 781c0beb96e8435a9dc711b0b70eb1da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:47:18.965566 3086 ts_tablet_manager.cc:1905] T 5e145751d0c145529dbf0449ecf5c8d4 P 781c0beb96e8435a9dc711b0b70eb1da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:47:18.967226 3085 tablet_replica.cc:330] T 5e145751d0c145529dbf0449ecf5c8d4 P 1750da561edc402883853661828d10c6: stopping tablet replica
I20260621 07:47:18.967415 3085 raft_consensus.cc:2229] T 5e145751d0c145529dbf0449ecf5c8d4 P 1750da561edc402883853661828d10c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:47:18.967574 3085 raft_consensus.cc:2258] T 5e145751d0c145529dbf0449ecf5c8d4 P 1750da561edc402883853661828d10c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:47:18.967657 3087 tablet_replica.cc:330] T 5e145751d0c145529dbf0449ecf5c8d4 P 1cd096c3570149679c0417ccd70dfe74: stopping tablet replica
I20260621 07:47:18.967830 3087 raft_consensus.cc:2229] T 5e145751d0c145529dbf0449ecf5c8d4 P 1cd096c3570149679c0417ccd70dfe74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:47:18.967970 3087 raft_consensus.cc:2258] T 5e145751d0c145529dbf0449ecf5c8d4 P 1cd096c3570149679c0417ccd70dfe74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:47:18.968243 3085 ts_tablet_manager.cc:1905] T 5e145751d0c145529dbf0449ecf5c8d4 P 1750da561edc402883853661828d10c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:47:18.968529 3087 ts_tablet_manager.cc:1905] T 5e145751d0c145529dbf0449ecf5c8d4 P 1cd096c3570149679c0417ccd70dfe74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:47:18.968756 3086 ts_tablet_manager.cc:1918] T 5e145751d0c145529dbf0449ecf5c8d4 P 781c0beb96e8435a9dc711b0b70eb1da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 07:47:18.968861 3086 log.cc:1196] T 5e145751d0c145529dbf0449ecf5c8d4 P 781c0beb96e8435a9dc711b0b70eb1da: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-1/wal/wals/5e145751d0c145529dbf0449ecf5c8d4
I20260621 07:47:18.969218 3086 ts_tablet_manager.cc:1939] T 5e145751d0c145529dbf0449ecf5c8d4 P 781c0beb96e8435a9dc711b0b70eb1da: Deleting consensus metadata
I20260621 07:47:18.969904 2486 catalog_manager.cc:4841] TS 781c0beb96e8435a9dc711b0b70eb1da (127.0.192.194:40397): tablet 5e145751d0c145529dbf0449ecf5c8d4 (table test-table [id=c3dc7c5fc356417da3632c84074ced3e]) successfully deleted
I20260621 07:47:18.970793 3085 ts_tablet_manager.cc:1918] T 5e145751d0c145529dbf0449ecf5c8d4 P 1750da561edc402883853661828d10c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 07:47:18.970893 3085 log.cc:1196] T 5e145751d0c145529dbf0449ecf5c8d4 P 1750da561edc402883853661828d10c6: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-0/wal/wals/5e145751d0c145529dbf0449ecf5c8d4
I20260621 07:47:18.971167 3085 ts_tablet_manager.cc:1939] T 5e145751d0c145529dbf0449ecf5c8d4 P 1750da561edc402883853661828d10c6: Deleting consensus metadata
I20260621 07:47:18.971529 3087 ts_tablet_manager.cc:1918] T 5e145751d0c145529dbf0449ecf5c8d4 P 1cd096c3570149679c0417ccd70dfe74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 07:47:18.971589 3087 log.cc:1196] T 5e145751d0c145529dbf0449ecf5c8d4 P 1cd096c3570149679c0417ccd70dfe74: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-2/wal/wals/5e145751d0c145529dbf0449ecf5c8d4
I20260621 07:47:18.971902 3087 ts_tablet_manager.cc:1939] T 5e145751d0c145529dbf0449ecf5c8d4 P 1cd096c3570149679c0417ccd70dfe74: Deleting consensus metadata
I20260621 07:47:18.971946 2487 catalog_manager.cc:4841] TS 1750da561edc402883853661828d10c6 (127.0.192.193:45443): tablet 5e145751d0c145529dbf0449ecf5c8d4 (table test-table [id=c3dc7c5fc356417da3632c84074ced3e]) successfully deleted
I20260621 07:47:18.972512 2487 catalog_manager.cc:4841] TS 1cd096c3570149679c0417ccd70dfe74 (127.0.192.195:36063): tablet 5e145751d0c145529dbf0449ecf5c8d4 (table test-table [id=c3dc7c5fc356417da3632c84074ced3e]) successfully deleted
Jun 21 07:47:18 dist-test-slave-lzx3 krb5kdc[2457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028038, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 07:47:18 dist-test-slave-lzx3 krb5kdc[2457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028038, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:47:18.989755 3095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:18.982239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58262 (local address 127.0.192.254:40179)
0621 07:47:18.982472 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:18.982478 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:18.982506 (+ 28us) server_negotiation.cc:407] Connection header received
0621 07:47:18.982606 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:18.982608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:18.982647 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:18.982715 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:18.983361 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:18.983890 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:18.984723 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:18.984920 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:18.984992 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:18.986997 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:18.987000 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:18.987002 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:18.987021 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:18.988210 (+ 1189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:18.988750 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:18.988754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:18.988756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:18.988844 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:18.989099 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:18.989101 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:18.989102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:18.989258 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:18.989356 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:18.989543 (+ 187us) server_negotiation.cc:299] Negotiation successful
0621 07:47:18.989639 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":108,"threads_started":1}
I20260621 07:47:18.992061 2496 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58262:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:47:18.992501 2496 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:47:18.996692 2639 tablet_service.cc:1467] Processing CreateTablet for tablet fcca14c83bef4cb89dde8b00e5402fb9 (DEFAULT_TABLE table=test-table [id=7c499d9b1d4e45fda0cd6c665a32e7b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:47:18.996771 2779 tablet_service.cc:1467] Processing CreateTablet for tablet fcca14c83bef4cb89dde8b00e5402fb9 (DEFAULT_TABLE table=test-table [id=7c499d9b1d4e45fda0cd6c665a32e7b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:47:18.996934 2918 tablet_service.cc:1467] Processing CreateTablet for tablet fcca14c83bef4cb89dde8b00e5402fb9 (DEFAULT_TABLE table=test-table [id=7c499d9b1d4e45fda0cd6c665a32e7b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:47:18.996986 2639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcca14c83bef4cb89dde8b00e5402fb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:18.997018 2779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcca14c83bef4cb89dde8b00e5402fb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:18.997157 2918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcca14c83bef4cb89dde8b00e5402fb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:18.998987 3096 tablet_bootstrap.cc:492] T fcca14c83bef4cb89dde8b00e5402fb9 P 1750da561edc402883853661828d10c6: Bootstrap starting.
I20260621 07:47:18.999665 3097 tablet_bootstrap.cc:492] T fcca14c83bef4cb89dde8b00e5402fb9 P 1cd096c3570149679c0417ccd70dfe74: Bootstrap starting.
I20260621 07:47:18.999872 3096 tablet_bootstrap.cc:654] T fcca14c83bef4cb89dde8b00e5402fb9 P 1750da561edc402883853661828d10c6: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:18.999939 3098 tablet_bootstrap.cc:492] T fcca14c83bef4cb89dde8b00e5402fb9 P 781c0beb96e8435a9dc711b0b70eb1da: Bootstrap starting.
I20260621 07:47:19.000450 3097 tablet_bootstrap.cc:654] T fcca14c83bef4cb89dde8b00e5402fb9 P 1cd096c3570149679c0417ccd70dfe74: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:19.000772 3098 tablet_bootstrap.cc:654] T fcca14c83bef4cb89dde8b00e5402fb9 P 781c0beb96e8435a9dc711b0b70eb1da: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:19.000954 3096 tablet_bootstrap.cc:492] T fcca14c83bef4cb89dde8b00e5402fb9 P 1750da561edc402883853661828d10c6: No bootstrap required, opened a new log
I20260621 07:47:19.001073 3096 ts_tablet_manager.cc:1397] T fcca14c83bef4cb89dde8b00e5402fb9 P 1750da561edc402883853661828d10c6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:19.001565 3097 tablet_bootstrap.cc:492] T fcca14c83bef4cb89dde8b00e5402fb9 P 1cd096c3570149679c0417ccd70dfe74: No bootstrap required, opened a new log
I20260621 07:47:19.001657 3096 raft_consensus.cc:348] T fcca14c83bef4cb89dde8b00e5402fb9 P 1750da561edc402883853661828d10c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } }
I20260621 07:47:19.001813 3097 ts_tablet_manager.cc:1397] T fcca14c83bef4cb89dde8b00e5402fb9 P 1cd096c3570149679c0417ccd70dfe74: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:19.001852 3098 tablet_bootstrap.cc:492] T fcca14c83bef4cb89dde8b00e5402fb9 P 781c0beb96e8435a9dc711b0b70eb1da: No bootstrap required, opened a new log
I20260621 07:47:19.001930 3098 ts_tablet_manager.cc:1397] T fcca14c83bef4cb89dde8b00e5402fb9 P 781c0beb96e8435a9dc711b0b70eb1da: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 07:47:19.001945 3096 raft_consensus.cc:374] T fcca14c83bef4cb89dde8b00e5402fb9 P 1750da561edc402883853661828d10c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:19.002058 3096 raft_consensus.cc:729] T fcca14c83bef4cb89dde8b00e5402fb9 P 1750da561edc402883853661828d10c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1750da561edc402883853661828d10c6, State: Initialized, Role: FOLLOWER
I20260621 07:47:19.002228 3096 consensus_queue.cc:260] T fcca14c83bef4cb89dde8b00e5402fb9 P 1750da561edc402883853661828d10c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } }
I20260621 07:47:19.002228 3097 raft_consensus.cc:348] T fcca14c83bef4cb89dde8b00e5402fb9 P 1cd096c3570149679c0417ccd70dfe74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } }
I20260621 07:47:19.002348 3097 raft_consensus.cc:374] T fcca14c83bef4cb89dde8b00e5402fb9 P 1cd096c3570149679c0417ccd70dfe74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:19.002331 3098 raft_consensus.cc:348] T fcca14c83bef4cb89dde8b00e5402fb9 P 781c0beb96e8435a9dc711b0b70eb1da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } }
I20260621 07:47:19.002408 3097 raft_consensus.cc:729] T fcca14c83bef4cb89dde8b00e5402fb9 P 1cd096c3570149679c0417ccd70dfe74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cd096c3570149679c0417ccd70dfe74, State: Initialized, Role: FOLLOWER
I20260621 07:47:19.002449 3098 raft_consensus.cc:374] T fcca14c83bef4cb89dde8b00e5402fb9 P 781c0beb96e8435a9dc711b0b70eb1da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:19.002488 3098 raft_consensus.cc:729] T fcca14c83bef4cb89dde8b00e5402fb9 P 781c0beb96e8435a9dc711b0b70eb1da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 781c0beb96e8435a9dc711b0b70eb1da, State: Initialized, Role: FOLLOWER
I20260621 07:47:19.002624 3096 ts_tablet_manager.cc:1428] T fcca14c83bef4cb89dde8b00e5402fb9 P 1750da561edc402883853661828d10c6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260621 07:47:19.002612 3098 consensus_queue.cc:260] T fcca14c83bef4cb89dde8b00e5402fb9 P 781c0beb96e8435a9dc711b0b70eb1da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } }
I20260621 07:47:19.002900 3098 ts_tablet_manager.cc:1428] T fcca14c83bef4cb89dde8b00e5402fb9 P 781c0beb96e8435a9dc711b0b70eb1da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260621 07:47:19.002555 3097 consensus_queue.cc:260] T fcca14c83bef4cb89dde8b00e5402fb9 P 1cd096c3570149679c0417ccd70dfe74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } }
I20260621 07:47:19.003582 3097 ts_tablet_manager.cc:1428] T fcca14c83bef4cb89dde8b00e5402fb9 P 1cd096c3570149679c0417ccd70dfe74: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20260621 07:47:19.004889 2846 tablet.cc:2369] T fcca14c83bef4cb89dde8b00e5402fb9 P 781c0beb96e8435a9dc711b0b70eb1da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:47:19.065582 3016 raft_consensus.cc:482] T fcca14c83bef4cb89dde8b00e5402fb9 P 781c0beb96e8435a9dc711b0b70eb1da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:47:19.065683 3016 raft_consensus.cc:504] T fcca14c83bef4cb89dde8b00e5402fb9 P 781c0beb96e8435a9dc711b0b70eb1da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } }
I20260621 07:47:19.066073 3016 leader_election.cc:288] T fcca14c83bef4cb89dde8b00e5402fb9 P 781c0beb96e8435a9dc711b0b70eb1da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1750da561edc402883853661828d10c6 (127.0.192.193:45443), 1cd096c3570149679c0417ccd70dfe74 (127.0.192.195:36063)
I20260621 07:47:19.066439 2659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcca14c83bef4cb89dde8b00e5402fb9" candidate_uuid: "781c0beb96e8435a9dc711b0b70eb1da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1750da561edc402883853661828d10c6" is_pre_election: true
I20260621 07:47:19.066473 2939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcca14c83bef4cb89dde8b00e5402fb9" candidate_uuid: "781c0beb96e8435a9dc711b0b70eb1da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cd096c3570149679c0417ccd70dfe74" is_pre_election: true
I20260621 07:47:19.066599 2659 raft_consensus.cc:2454] T fcca14c83bef4cb89dde8b00e5402fb9 P 1750da561edc402883853661828d10c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 781c0beb96e8435a9dc711b0b70eb1da in term 0.
I20260621 07:47:19.066625 2939 raft_consensus.cc:2454] T fcca14c83bef4cb89dde8b00e5402fb9 P 1cd096c3570149679c0417ccd70dfe74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 781c0beb96e8435a9dc711b0b70eb1da in term 0.
I20260621 07:47:19.066929 2731 leader_election.cc:302] T fcca14c83bef4cb89dde8b00e5402fb9 P 781c0beb96e8435a9dc711b0b70eb1da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cd096c3570149679c0417ccd70dfe74, 781c0beb96e8435a9dc711b0b70eb1da; no voters:
I20260621 07:47:19.067128 3016 raft_consensus.cc:2781] T fcca14c83bef4cb89dde8b00e5402fb9 P 781c0beb96e8435a9dc711b0b70eb1da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:47:19.067188 3016 raft_consensus.cc:482] T fcca14c83bef4cb89dde8b00e5402fb9 P 781c0beb96e8435a9dc711b0b70eb1da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:47:19.067232 3016 raft_consensus.cc:3037] T fcca14c83bef4cb89dde8b00e5402fb9 P 781c0beb96e8435a9dc711b0b70eb1da [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:19.068094 3016 raft_consensus.cc:504] T fcca14c83bef4cb89dde8b00e5402fb9 P 781c0beb96e8435a9dc711b0b70eb1da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } }
I20260621 07:47:19.068384 3016 leader_election.cc:288] T fcca14c83bef4cb89dde8b00e5402fb9 P 781c0beb96e8435a9dc711b0b70eb1da [CANDIDATE]: Term 1 election: Requested vote from peers 1750da561edc402883853661828d10c6 (127.0.192.193:45443), 1cd096c3570149679c0417ccd70dfe74 (127.0.192.195:36063)
I20260621 07:47:19.068723 2659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcca14c83bef4cb89dde8b00e5402fb9" candidate_uuid: "781c0beb96e8435a9dc711b0b70eb1da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1750da561edc402883853661828d10c6"
I20260621 07:47:19.068747 2939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcca14c83bef4cb89dde8b00e5402fb9" candidate_uuid: "781c0beb96e8435a9dc711b0b70eb1da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cd096c3570149679c0417ccd70dfe74"
I20260621 07:47:19.068852 2939 raft_consensus.cc:3037] T fcca14c83bef4cb89dde8b00e5402fb9 P 1cd096c3570149679c0417ccd70dfe74 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:19.068850 2659 raft_consensus.cc:3037] T fcca14c83bef4cb89dde8b00e5402fb9 P 1750da561edc402883853661828d10c6 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:19.069914 2939 raft_consensus.cc:2454] T fcca14c83bef4cb89dde8b00e5402fb9 P 1cd096c3570149679c0417ccd70dfe74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 781c0beb96e8435a9dc711b0b70eb1da in term 1.
I20260621 07:47:19.069914 2659 raft_consensus.cc:2454] T fcca14c83bef4cb89dde8b00e5402fb9 P 1750da561edc402883853661828d10c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 781c0beb96e8435a9dc711b0b70eb1da in term 1.
I20260621 07:47:19.070165 2731 leader_election.cc:302] T fcca14c83bef4cb89dde8b00e5402fb9 P 781c0beb96e8435a9dc711b0b70eb1da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cd096c3570149679c0417ccd70dfe74, 781c0beb96e8435a9dc711b0b70eb1da; no voters:
I20260621 07:47:19.070334 3016 raft_consensus.cc:2781] T fcca14c83bef4cb89dde8b00e5402fb9 P 781c0beb96e8435a9dc711b0b70eb1da [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:47:19.070421 3016 raft_consensus.cc:686] T fcca14c83bef4cb89dde8b00e5402fb9 P 781c0beb96e8435a9dc711b0b70eb1da [term 1 LEADER]: Becoming Leader. State: Replica: 781c0beb96e8435a9dc711b0b70eb1da, State: Running, Role: LEADER
I20260621 07:47:19.070585 3016 consensus_queue.cc:237] T fcca14c83bef4cb89dde8b00e5402fb9 P 781c0beb96e8435a9dc711b0b70eb1da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } }
I20260621 07:47:19.072038 2496 catalog_manager.cc:5495] T fcca14c83bef4cb89dde8b00e5402fb9 P 781c0beb96e8435a9dc711b0b70eb1da reported cstate change: term changed from 0 to 1, leader changed from <none> to 781c0beb96e8435a9dc711b0b70eb1da (127.0.192.194). New cstate: current_term: 1 leader_uuid: "781c0beb96e8435a9dc711b0b70eb1da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } health_report { overall_health: UNKNOWN } } }
I20260621 07:47:19.085564 3105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:19.082783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58988 (local address 127.0.192.194:40397)
0621 07:47:19.083032 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:19.083038 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:19.083050 (+ 12us) server_negotiation.cc:407] Connection header received
0621 07:47:19.083094 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:19.083096 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:19.083133 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:19.083200 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:47:19.083425 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:19.083919 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:19.084725 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:19.084929 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:19.084998 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:19.085190 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:47:19.085245 (+ 55us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:47:19.085292 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:19.085361 (+ 69us) server_negotiation.cc:299] Negotiation successful
0621 07:47:19.085397 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":90,"threads_started":1}
I20260621 07:47:19.090447 2659 raft_consensus.cc:1261] T fcca14c83bef4cb89dde8b00e5402fb9 P 1750da561edc402883853661828d10c6 [term 1 FOLLOWER]: Refusing update from remote peer 781c0beb96e8435a9dc711b0b70eb1da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:47:19.090452 2939 raft_consensus.cc:1261] T fcca14c83bef4cb89dde8b00e5402fb9 P 1cd096c3570149679c0417ccd70dfe74 [term 1 FOLLOWER]: Refusing update from remote peer 781c0beb96e8435a9dc711b0b70eb1da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:47:19.090783 3078 consensus_queue.cc:1035] T fcca14c83bef4cb89dde8b00e5402fb9 P 781c0beb96e8435a9dc711b0b70eb1da [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:47:19.090914 3016 consensus_queue.cc:1035] T fcca14c83bef4cb89dde8b00e5402fb9 P 781c0beb96e8435a9dc711b0b70eb1da [LEADER]: Connected to new peer: Peer: permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:47:19.117461 2496 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:47:19.117588 2496 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:47:19.119326 2496 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 147c497bf1a945a1a37de938664b2f47: Sending DeleteTablet for 3 replicas of tablet fcca14c83bef4cb89dde8b00e5402fb9
I20260621 07:47:19.119784 2639 tablet_service.cc:1514] Processing DeleteTablet for tablet fcca14c83bef4cb89dde8b00e5402fb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:47:19 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:58144
I20260621 07:47:19.119853 2918 tablet_service.cc:1514] Processing DeleteTablet for tablet fcca14c83bef4cb89dde8b00e5402fb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:47:19 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:51408
I20260621 07:47:19.119827 2779 tablet_service.cc:1514] Processing DeleteTablet for tablet fcca14c83bef4cb89dde8b00e5402fb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:47:19 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:58982
I20260621 07:47:19.119953 3085 tablet_replica.cc:330] T fcca14c83bef4cb89dde8b00e5402fb9 P 1750da561edc402883853661828d10c6: stopping tablet replica
I20260621 07:47:19.119979 3087 tablet_replica.cc:330] T fcca14c83bef4cb89dde8b00e5402fb9 P 1cd096c3570149679c0417ccd70dfe74: stopping tablet replica
I20260621 07:47:19.120045 3085 raft_consensus.cc:2229] T fcca14c83bef4cb89dde8b00e5402fb9 P 1750da561edc402883853661828d10c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:47:19.120081 3087 raft_consensus.cc:2229] T fcca14c83bef4cb89dde8b00e5402fb9 P 1cd096c3570149679c0417ccd70dfe74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:47:19.120142 3085 raft_consensus.cc:2258] T fcca14c83bef4cb89dde8b00e5402fb9 P 1750da561edc402883853661828d10c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:47:19.120157 3087 raft_consensus.cc:2258] T fcca14c83bef4cb89dde8b00e5402fb9 P 1cd096c3570149679c0417ccd70dfe74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:47:19.120285 3086 tablet_replica.cc:330] T fcca14c83bef4cb89dde8b00e5402fb9 P 781c0beb96e8435a9dc711b0b70eb1da: stopping tablet replica
I20260621 07:47:19.120388 3086 raft_consensus.cc:2229] T fcca14c83bef4cb89dde8b00e5402fb9 P 781c0beb96e8435a9dc711b0b70eb1da [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:47:19.120589 3086 raft_consensus.cc:2258] T fcca14c83bef4cb89dde8b00e5402fb9 P 781c0beb96e8435a9dc711b0b70eb1da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:47:19.120702 2496 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58262:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260621 07:47:19.120833 3087 ts_tablet_manager.cc:1905] T fcca14c83bef4cb89dde8b00e5402fb9 P 1cd096c3570149679c0417ccd70dfe74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:47:19.121201 3085 ts_tablet_manager.cc:1905] T fcca14c83bef4cb89dde8b00e5402fb9 P 1750da561edc402883853661828d10c6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260621 07:47:19.121203 2496 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:47:19.123312 3087 ts_tablet_manager.cc:1918] T fcca14c83bef4cb89dde8b00e5402fb9 P 1cd096c3570149679c0417ccd70dfe74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 07:47:19.123387 3087 log.cc:1196] T fcca14c83bef4cb89dde8b00e5402fb9 P 1cd096c3570149679c0417ccd70dfe74: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-2/wal/wals/fcca14c83bef4cb89dde8b00e5402fb9
I20260621 07:47:19.123519 3085 ts_tablet_manager.cc:1918] T fcca14c83bef4cb89dde8b00e5402fb9 P 1750da561edc402883853661828d10c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 07:47:19.123581 3085 log.cc:1196] T fcca14c83bef4cb89dde8b00e5402fb9 P 1750da561edc402883853661828d10c6: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-0/wal/wals/fcca14c83bef4cb89dde8b00e5402fb9
I20260621 07:47:19.123644 3087 ts_tablet_manager.cc:1939] T fcca14c83bef4cb89dde8b00e5402fb9 P 1cd096c3570149679c0417ccd70dfe74: Deleting consensus metadata
I20260621 07:47:19.123764 3085 ts_tablet_manager.cc:1939] T fcca14c83bef4cb89dde8b00e5402fb9 P 1750da561edc402883853661828d10c6: Deleting consensus metadata
I20260621 07:47:19.124293 3086 ts_tablet_manager.cc:1905] T fcca14c83bef4cb89dde8b00e5402fb9 P 781c0beb96e8435a9dc711b0b70eb1da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:47:19.124298 2487 catalog_manager.cc:4841] TS 1cd096c3570149679c0417ccd70dfe74 (127.0.192.195:36063): tablet fcca14c83bef4cb89dde8b00e5402fb9 (table test-table [id=7c499d9b1d4e45fda0cd6c665a32e7b4]) successfully deleted
I20260621 07:47:19.125319 2487 catalog_manager.cc:4841] TS 1750da561edc402883853661828d10c6 (127.0.192.193:45443): tablet fcca14c83bef4cb89dde8b00e5402fb9 (table test-table [id=7c499d9b1d4e45fda0cd6c665a32e7b4]) successfully deleted
I20260621 07:47:19.125852 2779 tablet_service.cc:1467] Processing CreateTablet for tablet a19b9c18fff1431fa934ea20111e137d (DEFAULT_TABLE table=test-table [id=53990e0b17044b74840a67de20275853]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:47:19.125860 2639 tablet_service.cc:1467] Processing CreateTablet for tablet a19b9c18fff1431fa934ea20111e137d (DEFAULT_TABLE table=test-table [id=53990e0b17044b74840a67de20275853]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:47:19.125948 2918 tablet_service.cc:1467] Processing CreateTablet for tablet a19b9c18fff1431fa934ea20111e137d (DEFAULT_TABLE table=test-table [id=53990e0b17044b74840a67de20275853]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:47:19.126055 2639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a19b9c18fff1431fa934ea20111e137d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:19.126067 2779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a19b9c18fff1431fa934ea20111e137d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:19.126148 2918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a19b9c18fff1431fa934ea20111e137d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:19.126510 3086 ts_tablet_manager.cc:1918] T fcca14c83bef4cb89dde8b00e5402fb9 P 781c0beb96e8435a9dc711b0b70eb1da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 07:47:19.126565 3086 log.cc:1196] T fcca14c83bef4cb89dde8b00e5402fb9 P 781c0beb96e8435a9dc711b0b70eb1da: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-1/wal/wals/fcca14c83bef4cb89dde8b00e5402fb9
I20260621 07:47:19.126770 3086 ts_tablet_manager.cc:1939] T fcca14c83bef4cb89dde8b00e5402fb9 P 781c0beb96e8435a9dc711b0b70eb1da: Deleting consensus metadata
I20260621 07:47:19.127341 2486 catalog_manager.cc:4841] TS 781c0beb96e8435a9dc711b0b70eb1da (127.0.192.194:40397): tablet fcca14c83bef4cb89dde8b00e5402fb9 (table test-table [id=7c499d9b1d4e45fda0cd6c665a32e7b4]) successfully deleted
I20260621 07:47:19.128037 3098 tablet_bootstrap.cc:492] T a19b9c18fff1431fa934ea20111e137d P 781c0beb96e8435a9dc711b0b70eb1da: Bootstrap starting.
I20260621 07:47:19.128058 3097 tablet_bootstrap.cc:492] T a19b9c18fff1431fa934ea20111e137d P 1cd096c3570149679c0417ccd70dfe74: Bootstrap starting.
I20260621 07:47:19.128227 3096 tablet_bootstrap.cc:492] T a19b9c18fff1431fa934ea20111e137d P 1750da561edc402883853661828d10c6: Bootstrap starting.
I20260621 07:47:19.128757 3097 tablet_bootstrap.cc:654] T a19b9c18fff1431fa934ea20111e137d P 1cd096c3570149679c0417ccd70dfe74: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:19.128837 3098 tablet_bootstrap.cc:654] T a19b9c18fff1431fa934ea20111e137d P 781c0beb96e8435a9dc711b0b70eb1da: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:19.129290 3096 tablet_bootstrap.cc:654] T a19b9c18fff1431fa934ea20111e137d P 1750da561edc402883853661828d10c6: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:19.130023 3098 tablet_bootstrap.cc:492] T a19b9c18fff1431fa934ea20111e137d P 781c0beb96e8435a9dc711b0b70eb1da: No bootstrap required, opened a new log
I20260621 07:47:19.130133 3098 ts_tablet_manager.cc:1397] T a19b9c18fff1431fa934ea20111e137d P 781c0beb96e8435a9dc711b0b70eb1da: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:19.130451 3097 tablet_bootstrap.cc:492] T a19b9c18fff1431fa934ea20111e137d P 1cd096c3570149679c0417ccd70dfe74: No bootstrap required, opened a new log
I20260621 07:47:19.130487 3098 raft_consensus.cc:348] T a19b9c18fff1431fa934ea20111e137d P 781c0beb96e8435a9dc711b0b70eb1da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } }
I20260621 07:47:19.130537 3097 ts_tablet_manager.cc:1397] T a19b9c18fff1431fa934ea20111e137d P 1cd096c3570149679c0417ccd70dfe74: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260621 07:47:19.130584 3098 raft_consensus.cc:374] T a19b9c18fff1431fa934ea20111e137d P 781c0beb96e8435a9dc711b0b70eb1da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:19.130638 3098 raft_consensus.cc:729] T a19b9c18fff1431fa934ea20111e137d P 781c0beb96e8435a9dc711b0b70eb1da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 781c0beb96e8435a9dc711b0b70eb1da, State: Initialized, Role: FOLLOWER
I20260621 07:47:19.130748 3098 consensus_queue.cc:260] T a19b9c18fff1431fa934ea20111e137d P 781c0beb96e8435a9dc711b0b70eb1da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } }
I20260621 07:47:19.130811 3096 tablet_bootstrap.cc:492] T a19b9c18fff1431fa934ea20111e137d P 1750da561edc402883853661828d10c6: No bootstrap required, opened a new log
I20260621 07:47:19.130913 3096 ts_tablet_manager.cc:1397] T a19b9c18fff1431fa934ea20111e137d P 1750da561edc402883853661828d10c6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260621 07:47:19.130913 3097 raft_consensus.cc:348] T a19b9c18fff1431fa934ea20111e137d P 1cd096c3570149679c0417ccd70dfe74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } }
I20260621 07:47:19.130995 3098 ts_tablet_manager.cc:1428] T a19b9c18fff1431fa934ea20111e137d P 781c0beb96e8435a9dc711b0b70eb1da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260621 07:47:19.131001 3097 raft_consensus.cc:374] T a19b9c18fff1431fa934ea20111e137d P 1cd096c3570149679c0417ccd70dfe74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:19.131032 3097 raft_consensus.cc:729] T a19b9c18fff1431fa934ea20111e137d P 1cd096c3570149679c0417ccd70dfe74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cd096c3570149679c0417ccd70dfe74, State: Initialized, Role: FOLLOWER
I20260621 07:47:19.131145 3097 consensus_queue.cc:260] T a19b9c18fff1431fa934ea20111e137d P 1cd096c3570149679c0417ccd70dfe74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } }
I20260621 07:47:19.131289 3096 raft_consensus.cc:348] T a19b9c18fff1431fa934ea20111e137d P 1750da561edc402883853661828d10c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } }
I20260621 07:47:19.131369 3097 ts_tablet_manager.cc:1428] T a19b9c18fff1431fa934ea20111e137d P 1cd096c3570149679c0417ccd70dfe74: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260621 07:47:19.131395 3096 raft_consensus.cc:374] T a19b9c18fff1431fa934ea20111e137d P 1750da561edc402883853661828d10c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:19.131428 3096 raft_consensus.cc:729] T a19b9c18fff1431fa934ea20111e137d P 1750da561edc402883853661828d10c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1750da561edc402883853661828d10c6, State: Initialized, Role: FOLLOWER
I20260621 07:47:19.131567 3096 consensus_queue.cc:260] T a19b9c18fff1431fa934ea20111e137d P 1750da561edc402883853661828d10c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } }
I20260621 07:47:19.131839 3096 ts_tablet_manager.cc:1428] T a19b9c18fff1431fa934ea20111e137d P 1750da561edc402883853661828d10c6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260621 07:47:19.179199 3078 raft_consensus.cc:482] T a19b9c18fff1431fa934ea20111e137d P 781c0beb96e8435a9dc711b0b70eb1da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:47:19.179299 3078 raft_consensus.cc:504] T a19b9c18fff1431fa934ea20111e137d P 781c0beb96e8435a9dc711b0b70eb1da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } }
I20260621 07:47:19.179730 3078 leader_election.cc:288] T a19b9c18fff1431fa934ea20111e137d P 781c0beb96e8435a9dc711b0b70eb1da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1750da561edc402883853661828d10c6 (127.0.192.193:45443), 1cd096c3570149679c0417ccd70dfe74 (127.0.192.195:36063)
I20260621 07:47:19.180130 2939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19b9c18fff1431fa934ea20111e137d" candidate_uuid: "781c0beb96e8435a9dc711b0b70eb1da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cd096c3570149679c0417ccd70dfe74" is_pre_election: true
I20260621 07:47:19.180127 2659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19b9c18fff1431fa934ea20111e137d" candidate_uuid: "781c0beb96e8435a9dc711b0b70eb1da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1750da561edc402883853661828d10c6" is_pre_election: true
I20260621 07:47:19.180258 2939 raft_consensus.cc:2454] T a19b9c18fff1431fa934ea20111e137d P 1cd096c3570149679c0417ccd70dfe74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 781c0beb96e8435a9dc711b0b70eb1da in term 0.
I20260621 07:47:19.180258 2659 raft_consensus.cc:2454] T a19b9c18fff1431fa934ea20111e137d P 1750da561edc402883853661828d10c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 781c0beb96e8435a9dc711b0b70eb1da in term 0.
I20260621 07:47:19.180526 2731 leader_election.cc:302] T a19b9c18fff1431fa934ea20111e137d P 781c0beb96e8435a9dc711b0b70eb1da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cd096c3570149679c0417ccd70dfe74, 781c0beb96e8435a9dc711b0b70eb1da; no voters:
I20260621 07:47:19.180718 3078 raft_consensus.cc:2781] T a19b9c18fff1431fa934ea20111e137d P 781c0beb96e8435a9dc711b0b70eb1da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:47:19.180791 3078 raft_consensus.cc:482] T a19b9c18fff1431fa934ea20111e137d P 781c0beb96e8435a9dc711b0b70eb1da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:47:19.180853 3078 raft_consensus.cc:3037] T a19b9c18fff1431fa934ea20111e137d P 781c0beb96e8435a9dc711b0b70eb1da [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:19.181726 3078 raft_consensus.cc:504] T a19b9c18fff1431fa934ea20111e137d P 781c0beb96e8435a9dc711b0b70eb1da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } }
I20260621 07:47:19.182044 3078 leader_election.cc:288] T a19b9c18fff1431fa934ea20111e137d P 781c0beb96e8435a9dc711b0b70eb1da [CANDIDATE]: Term 1 election: Requested vote from peers 1750da561edc402883853661828d10c6 (127.0.192.193:45443), 1cd096c3570149679c0417ccd70dfe74 (127.0.192.195:36063)
I20260621 07:47:19.182322 2659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19b9c18fff1431fa934ea20111e137d" candidate_uuid: "781c0beb96e8435a9dc711b0b70eb1da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1750da561edc402883853661828d10c6"
I20260621 07:47:19.182332 2939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19b9c18fff1431fa934ea20111e137d" candidate_uuid: "781c0beb96e8435a9dc711b0b70eb1da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cd096c3570149679c0417ccd70dfe74"
I20260621 07:47:19.182405 2659 raft_consensus.cc:3037] T a19b9c18fff1431fa934ea20111e137d P 1750da561edc402883853661828d10c6 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:19.182456 2939 raft_consensus.cc:3037] T a19b9c18fff1431fa934ea20111e137d P 1cd096c3570149679c0417ccd70dfe74 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:19.183220 2659 raft_consensus.cc:2454] T a19b9c18fff1431fa934ea20111e137d P 1750da561edc402883853661828d10c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 781c0beb96e8435a9dc711b0b70eb1da in term 1.
I20260621 07:47:19.183372 2939 raft_consensus.cc:2454] T a19b9c18fff1431fa934ea20111e137d P 1cd096c3570149679c0417ccd70dfe74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 781c0beb96e8435a9dc711b0b70eb1da in term 1.
I20260621 07:47:19.183506 2731 leader_election.cc:302] T a19b9c18fff1431fa934ea20111e137d P 781c0beb96e8435a9dc711b0b70eb1da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1750da561edc402883853661828d10c6, 781c0beb96e8435a9dc711b0b70eb1da; no voters:
I20260621 07:47:19.183686 3078 raft_consensus.cc:2781] T a19b9c18fff1431fa934ea20111e137d P 781c0beb96e8435a9dc711b0b70eb1da [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:47:19.183780 3078 raft_consensus.cc:686] T a19b9c18fff1431fa934ea20111e137d P 781c0beb96e8435a9dc711b0b70eb1da [term 1 LEADER]: Becoming Leader. State: Replica: 781c0beb96e8435a9dc711b0b70eb1da, State: Running, Role: LEADER
I20260621 07:47:19.183940 3078 consensus_queue.cc:237] T a19b9c18fff1431fa934ea20111e137d P 781c0beb96e8435a9dc711b0b70eb1da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } }
I20260621 07:47:19.185387 2502 catalog_manager.cc:5495] T a19b9c18fff1431fa934ea20111e137d P 781c0beb96e8435a9dc711b0b70eb1da reported cstate change: term changed from 0 to 1, leader changed from <none> to 781c0beb96e8435a9dc711b0b70eb1da (127.0.192.194). New cstate: current_term: 1 leader_uuid: "781c0beb96e8435a9dc711b0b70eb1da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "781c0beb96e8435a9dc711b0b70eb1da" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 } health_report { overall_health: UNKNOWN } } }
I20260621 07:47:19.202553 2659 raft_consensus.cc:1261] T a19b9c18fff1431fa934ea20111e137d P 1750da561edc402883853661828d10c6 [term 1 FOLLOWER]: Refusing update from remote peer 781c0beb96e8435a9dc711b0b70eb1da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:47:19.202581 2939 raft_consensus.cc:1261] T a19b9c18fff1431fa934ea20111e137d P 1cd096c3570149679c0417ccd70dfe74 [term 1 FOLLOWER]: Refusing update from remote peer 781c0beb96e8435a9dc711b0b70eb1da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:47:19.202970 3078 consensus_queue.cc:1035] T a19b9c18fff1431fa934ea20111e137d P 781c0beb96e8435a9dc711b0b70eb1da [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd096c3570149679c0417ccd70dfe74" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 36063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:47:19.203083 3077 consensus_queue.cc:1035] T a19b9c18fff1431fa934ea20111e137d P 781c0beb96e8435a9dc711b0b70eb1da [LEADER]: Connected to new peer: Peer: permanent_uuid: "1750da561edc402883853661828d10c6" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260621 07:47:19.215754 3068 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260621 07:47:20.763448 2496 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:47:20.763602 2496 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:47:20.766014 2496 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 147c497bf1a945a1a37de938664b2f47: Sending DeleteTablet for 3 replicas of tablet a19b9c18fff1431fa934ea20111e137d
I20260621 07:47:20.766557 2779 tablet_service.cc:1514] Processing DeleteTablet for tablet a19b9c18fff1431fa934ea20111e137d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:47:20 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:58982
I20260621 07:47:20.766587 2918 tablet_service.cc:1514] Processing DeleteTablet for tablet a19b9c18fff1431fa934ea20111e137d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:47:20 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:51408
I20260621 07:47:20.766582 2639 tablet_service.cc:1514] Processing DeleteTablet for tablet a19b9c18fff1431fa934ea20111e137d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:47:20 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:58144
I20260621 07:47:20.766863 3139 tablet_replica.cc:330] T a19b9c18fff1431fa934ea20111e137d P 781c0beb96e8435a9dc711b0b70eb1da: stopping tablet replica
I20260621 07:47:20.766974 3139 raft_consensus.cc:2229] T a19b9c18fff1431fa934ea20111e137d P 781c0beb96e8435a9dc711b0b70eb1da [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:47:20.767031 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 2574
I20260621 07:47:20.767005 3140 tablet_replica.cc:330] T a19b9c18fff1431fa934ea20111e137d P 1750da561edc402883853661828d10c6: stopping tablet replica
I20260621 07:47:20.767139 3140 raft_consensus.cc:2229] T a19b9c18fff1431fa934ea20111e137d P 1750da561edc402883853661828d10c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:47:20.767642 3139 raft_consensus.cc:2258] T a19b9c18fff1431fa934ea20111e137d P 781c0beb96e8435a9dc711b0b70eb1da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:47:20.768151 3139 ts_tablet_manager.cc:1905] T a19b9c18fff1431fa934ea20111e137d P 781c0beb96e8435a9dc711b0b70eb1da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:47:20.770157 3138 tablet_replica.cc:330] T a19b9c18fff1431fa934ea20111e137d P 1cd096c3570149679c0417ccd70dfe74: stopping tablet replica
I20260621 07:47:20.770315 3138 raft_consensus.cc:2229] T a19b9c18fff1431fa934ea20111e137d P 1cd096c3570149679c0417ccd70dfe74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:47:20.770335 3139 ts_tablet_manager.cc:1918] T a19b9c18fff1431fa934ea20111e137d P 781c0beb96e8435a9dc711b0b70eb1da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 07:47:20.770408 3139 log.cc:1196] T a19b9c18fff1431fa934ea20111e137d P 781c0beb96e8435a9dc711b0b70eb1da: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-1/wal/wals/a19b9c18fff1431fa934ea20111e137d
I20260621 07:47:20.770442 3138 raft_consensus.cc:2258] T a19b9c18fff1431fa934ea20111e137d P 1cd096c3570149679c0417ccd70dfe74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:47:20.770720 3139 ts_tablet_manager.cc:1939] T a19b9c18fff1431fa934ea20111e137d P 781c0beb96e8435a9dc711b0b70eb1da: Deleting consensus metadata
I20260621 07:47:20.770948 3138 ts_tablet_manager.cc:1905] T a19b9c18fff1431fa934ea20111e137d P 1cd096c3570149679c0417ccd70dfe74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:47:20.771436 2486 catalog_manager.cc:4841] TS 781c0beb96e8435a9dc711b0b70eb1da (127.0.192.194:40397): tablet a19b9c18fff1431fa934ea20111e137d (table test-table [id=53990e0b17044b74840a67de20275853]) successfully deleted
I20260621 07:47:20.772998 3138 ts_tablet_manager.cc:1918] T a19b9c18fff1431fa934ea20111e137d P 1cd096c3570149679c0417ccd70dfe74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 07:47:20.773097 3138 log.cc:1196] T a19b9c18fff1431fa934ea20111e137d P 1cd096c3570149679c0417ccd70dfe74: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782028028522580-771-0/minicluster-data/ts-2/wal/wals/a19b9c18fff1431fa934ea20111e137d
I20260621 07:47:20.773360 3138 ts_tablet_manager.cc:1939] T a19b9c18fff1431fa934ea20111e137d P 1cd096c3570149679c0417ccd70dfe74: Deleting consensus metadata
W20260621 07:47:20.773499 2487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.192.193:45443 (error 108)
I20260621 07:47:20.774024 2487 catalog_manager.cc:4841] TS 1cd096c3570149679c0417ccd70dfe74 (127.0.192.195:36063): tablet a19b9c18fff1431fa934ea20111e137d (table test-table [id=53990e0b17044b74840a67de20275853]) successfully deleted
I20260621 07:47:20.774336 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 2714
I20260621 07:47:20.774968 3142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:20.774295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:45443 (local address 127.0.0.1:36746)
0621 07:47:20.774535 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:20.774866 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.193:45443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":96,"threads_started":1}
W20260621 07:47:20.775645 2487 catalog_manager.cc:4584] TS 1750da561edc402883853661828d10c6 (127.0.192.193:45443): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a19b9c18fff1431fa934ea20111e137d: Network error: Client connection negotiation failed: client connection to 127.0.192.193:45443: connect: Connection refused (error 111)
I20260621 07:47:20.781054 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 2854
I20260621 07:47:20.787169 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 2470
2026-06-21T07:47:20Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5686 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:47:20 dist-test-slave-lzx3 krb5kdc[3145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:47:20 dist-test-slave-lzx3 krb5kdc[3145](info): set up 2 sockets
Jun 21 07:47:20 dist-test-slave-lzx3 krb5kdc[3145](info): commencing operation
krb5kdc: starting...
W20260621 07:47:21.768612 771 mini_kdc.cc:121] Time spent starting KDC: real 0.960s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/krb5kdc/test-user.keytab.
Jun 21 07:47:21 dist-test-slave-lzx3 krb5kdc[3145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:47:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:47:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.192.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
I20260621 07:47:21.879704 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:35445
--webserver_interface=127.0.192.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:40501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:35445
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:21.981217 3158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:21.981446 3158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:21.981480 3158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:21.985320 3158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:47:21.985359 3158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:21.985376 3158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:21.985395 3158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:47:21.985407 3158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:47:21.989962 3158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:40501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:35445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:35445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:21.990743 3158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:21.991551 3158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:21.999271 3164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:21.999402 3158 server_base.cc:1034] running on GCE node
W20260621 07:47:21.999366 3163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:21.999344 3166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:22.000043 3158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:22.001036 3158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:22.002187 3158 hybrid_clock.cc:648] HybridClock initialized: now 1782028042002169 us; error 32 us; skew 500 ppm
Jun 21 07:47:21 dist-test-slave-lzx3 krb5kdc[3145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028041, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:22.004832 3158 init.cc:377] Logged in from keytab as kudu/127.0.192.254@KRBTEST.COM (short username kudu)
I20260621 07:47:22.005613 3158 webserver.cc:460] Webserver started at http://127.0.192.254:37905/ using document root <none> and password file <none>
I20260621 07:47:22.006109 3158 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:22.006212 3158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:22.006412 3158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:22.008111 3158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "fbf401214c7248368b064be088e1d1b7"
format_stamp: "Formatted at 2026-06-21 07:47:22 on dist-test-slave-lzx3"
server_key: "8ad95550eee4b6b91f7981cfcc9f1f9a"
server_key_iv: "160aeffab6e27f5026bb091354b93223"
server_key_version: "encryptionkey@0"
I20260621 07:47:22.008677 3158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "fbf401214c7248368b064be088e1d1b7"
format_stamp: "Formatted at 2026-06-21 07:47:22 on dist-test-slave-lzx3"
server_key: "8ad95550eee4b6b91f7981cfcc9f1f9a"
server_key_iv: "160aeffab6e27f5026bb091354b93223"
server_key_version: "encryptionkey@0"
I20260621 07:47:22.011791 3158 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260621 07:47:22.014204 3173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:22.015337 3158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 07:47:22.015475 3158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "fbf401214c7248368b064be088e1d1b7"
format_stamp: "Formatted at 2026-06-21 07:47:22 on dist-test-slave-lzx3"
server_key: "8ad95550eee4b6b91f7981cfcc9f1f9a"
server_key_iv: "160aeffab6e27f5026bb091354b93223"
server_key_version: "encryptionkey@0"
I20260621 07:47:22.015578 3158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:22.026077 3158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:22.029063 3158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:22.029248 3158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:22.035705 3158 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:35445
I20260621 07:47:22.035744 3225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:35445 every 8 connection(s)
I20260621 07:47:22.036628 3158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:47:22.039028 3226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:22.043699 3226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbf401214c7248368b064be088e1d1b7: Bootstrap starting.
I20260621 07:47:22.044652 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 3158
I20260621 07:47:22.044747 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:47:22.045001 771 external_mini_cluster.cc:1428] Setting key a0f37f7ac4ce9c933553abe5e6b535b0
I20260621 07:47:22.046229 3226 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbf401214c7248368b064be088e1d1b7: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:22.046886 3226 log.cc:824] T 00000000000000000000000000000000 P fbf401214c7248368b064be088e1d1b7: Log is configured to *not* fsync() on all Append() calls
I20260621 07:47:22.048897 3226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbf401214c7248368b064be088e1d1b7: No bootstrap required, opened a new log
I20260621 07:47:22.053013 3226 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbf401214c7248368b064be088e1d1b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf401214c7248368b064be088e1d1b7" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 35445 } }
Jun 21 07:47:22 dist-test-slave-lzx3 krb5kdc[3145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:47:22.053241 3226 raft_consensus.cc:374] T 00000000000000000000000000000000 P fbf401214c7248368b064be088e1d1b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:22.053328 3226 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbf401214c7248368b064be088e1d1b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbf401214c7248368b064be088e1d1b7, State: Initialized, Role: FOLLOWER
I20260621 07:47:22.053735 3226 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbf401214c7248368b064be088e1d1b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf401214c7248368b064be088e1d1b7" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 35445 } }
I20260621 07:47:22.053885 3226 raft_consensus.cc:388] T 00000000000000000000000000000000 P fbf401214c7248368b064be088e1d1b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:47:22.053979 3226 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbf401214c7248368b064be088e1d1b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:47:22.054098 3226 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbf401214c7248368b064be088e1d1b7 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:22.055107 3226 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbf401214c7248368b064be088e1d1b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf401214c7248368b064be088e1d1b7" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 35445 } }
I20260621 07:47:22.055421 3226 leader_election.cc:302] T 00000000000000000000000000000000 P fbf401214c7248368b064be088e1d1b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fbf401214c7248368b064be088e1d1b7; no voters:
I20260621 07:47:22.055719 3226 leader_election.cc:288] T 00000000000000000000000000000000 P fbf401214c7248368b064be088e1d1b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:47:22.055778 3231 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fbf401214c7248368b064be088e1d1b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:47:22.055938 3231 raft_consensus.cc:686] T 00000000000000000000000000000000 P fbf401214c7248368b064be088e1d1b7 [term 1 LEADER]: Becoming Leader. State: Replica: fbf401214c7248368b064be088e1d1b7, State: Running, Role: LEADER
I20260621 07:47:22.056264 3231 consensus_queue.cc:237] T 00000000000000000000000000000000 P fbf401214c7248368b064be088e1d1b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf401214c7248368b064be088e1d1b7" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 35445 } }
I20260621 07:47:22.056896 3226 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbf401214c7248368b064be088e1d1b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:47:22.058077 3233 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbf401214c7248368b064be088e1d1b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbf401214c7248368b064be088e1d1b7. Latest consensus state: current_term: 1 leader_uuid: "fbf401214c7248368b064be088e1d1b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf401214c7248368b064be088e1d1b7" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 35445 } } }
I20260621 07:47:22.058187 3233 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbf401214c7248368b064be088e1d1b7 [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:22.058986 3229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:22.045982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52786 (local address 127.0.192.254:35445)
0621 07:47:22.046372 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:22.046390 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:22.046429 (+ 39us) server_negotiation.cc:407] Connection header received
0621 07:47:22.046947 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:22.046960 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:22.047332 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:22.047636 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:22.048448 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:22.049405 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:22.050595 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:22.051020 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:22.051127 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:22.053899 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:22.053906 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:22.053922 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:22.053953 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:22.055748 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:22.056246 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:22.056252 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:22.056256 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:22.056406 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:22.056718 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:22.056721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:22.056723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:22.057143 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:22.057370 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:22.057613 (+ 243us) server_negotiation.cc:299] Negotiation successful
0621 07:47:22.057922 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":109,"threads_started":1}
I20260621 07:47:22.059084 3232 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbf401214c7248368b064be088e1d1b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fbf401214c7248368b064be088e1d1b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf401214c7248368b064be088e1d1b7" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 35445 } } }
I20260621 07:47:22.059201 3232 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbf401214c7248368b064be088e1d1b7 [sys.catalog]: This master's current role is: LEADER
W20260621 07:47:22.062389 3246 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fbf401214c7248368b064be088e1d1b7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:47:22.062476 3246 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:47:22.062561 3241 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:47:22.063218 3241 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:47:22.068372 3241 catalog_manager.cc:1347] Generated new cluster ID: e6a42f7023b34297a420b1e9bdccd381
I20260621 07:47:22.068432 3241 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:47:22.082086 3241 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:47:22.083087 3241 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:47:22.094504 3241 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fbf401214c7248368b064be088e1d1b7: Generated new TSK 0
I20260621 07:47:22.095155 3241 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.192.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab.
Entry for principal kudu/127.0.192.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab.
I20260621 07:47:22.120103 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.193:0
--local_ip_for_outbound_sockets=127.0.192.193
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:35445
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:40501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:22.222716 3252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:22.222909 3252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:22.222944 3252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:22.227092 3252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:22.227141 3252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:22.227246 3252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.193
I20260621 07:47:22.232178 3252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:40501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:35445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.193
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:22.233146 3252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:22.234045 3252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:22.242770 3260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:22.242780 3258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:22.243031 3252 server_base.cc:1034] running on GCE node
W20260621 07:47:22.242857 3257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:22.243438 3252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:22.244004 3252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:22.245184 3252 hybrid_clock.cc:648] HybridClock initialized: now 1782028042245150 us; error 47 us; skew 500 ppm
Jun 21 07:47:22 dist-test-slave-lzx3 krb5kdc[3145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028042, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:22.247536 3252 init.cc:377] Logged in from keytab as kudu/127.0.192.193@KRBTEST.COM (short username kudu)
I20260621 07:47:22.248261 3252 webserver.cc:460] Webserver started at http://127.0.192.193:33317/ using document root <none> and password file <none>
I20260621 07:47:22.248746 3252 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:22.248798 3252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:22.248943 3252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:22.250492 3252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-0/data/instance:
uuid: "df499f2ee4dd448c8edc34aad9d813f4"
format_stamp: "Formatted at 2026-06-21 07:47:22 on dist-test-slave-lzx3"
server_key: "329c56d9db21fc0b89ea76d161d54884"
server_key_iv: "6e607d3b9f742cf1b1658e203c7f0719"
server_key_version: "encryptionkey@0"
I20260621 07:47:22.250912 3252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-0/wal/instance:
uuid: "df499f2ee4dd448c8edc34aad9d813f4"
format_stamp: "Formatted at 2026-06-21 07:47:22 on dist-test-slave-lzx3"
server_key: "329c56d9db21fc0b89ea76d161d54884"
server_key_iv: "6e607d3b9f742cf1b1658e203c7f0719"
server_key_version: "encryptionkey@0"
I20260621 07:47:22.254025 3252 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 07:47:22.256094 3267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:22.257262 3252 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:22.257340 3252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-0/wal
uuid: "df499f2ee4dd448c8edc34aad9d813f4"
format_stamp: "Formatted at 2026-06-21 07:47:22 on dist-test-slave-lzx3"
server_key: "329c56d9db21fc0b89ea76d161d54884"
server_key_iv: "6e607d3b9f742cf1b1658e203c7f0719"
server_key_version: "encryptionkey@0"
I20260621 07:47:22.257426 3252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:22.271690 3252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:22.274312 3252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:22.274513 3252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:22.275066 3252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:47:22.275720 3252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:47:22.275774 3252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:22.275807 3252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:47:22.275820 3252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:22.284533 3252 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.193:33635
I20260621 07:47:22.284646 3380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.193:33635 every 8 connection(s)
I20260621 07:47:22.285403 3252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
Jun 21 07:47:22 dist-test-slave-lzx3 krb5kdc[3145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028042, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:47:22.295063 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 3252
I20260621 07:47:22.295197 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-0/wal/instance
I20260621 07:47:22.295449 771 external_mini_cluster.cc:1428] Setting key 18b67cf3f10bd621a3c05cfb4bff62ae
I20260621 07:47:22.297719 3229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:22.287024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:35679 (local address 127.0.192.254:35445)
0621 07:47:22.287159 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:22.287162 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:22.287939 (+ 777us) server_negotiation.cc:407] Connection header received
0621 07:47:22.289263 (+ 1324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:22.289265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:22.289308 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:22.289378 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:22.290413 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:22.290927 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:22.291927 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:22.292141 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:22.292207 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:22.294380 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:22.294384 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:22.294387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:22.294413 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:22.295787 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:22.296240 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:22.296243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:22.296245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:22.296339 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:22.296715 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:22.296718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:22.296721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:22.296922 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:22.296997 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:22.297529 (+ 532us) server_negotiation.cc:299] Negotiation successful
0621 07:47:22.297614 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 07:47:22.298733 3383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:22.287340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:35445 (local address 127.0.192.193:35679)
0621 07:47:22.287784 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:22.287822 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:22.289079 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:22.289535 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:22.289540 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:22.289847 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:22.290289 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:22.290300 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:22.291024 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:22.291026 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:22.291825 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:22.291833 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:22.292249 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:22.292251 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:22.292365 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:22.292876 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:22.292890 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:22.294222 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:22.295866 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:22.295871 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:22.295884 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:22.296156 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:22.296426 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:22.296429 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:22.296430 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:22.296603 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:22.297027 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:22.297034 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:22.297301 (+ 267us) client_negotiation.cc:769] Sending connection context
0621 07:47:22.297481 (+ 180us) client_negotiation.cc:240] Negotiation successful
0621 07:47:22.297791 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":128,"threads_started":1}
I20260621 07:47:22.299857 3381 heartbeater.cc:347] Connected to a master server at 127.0.192.254:35445
I20260621 07:47:22.300076 3381 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:22.300495 3381 heartbeater.cc:511] Master 127.0.192.254:35445 requested a full tablet report, sending...
I20260621 07:47:22.301923 3190 ts_manager.cc:194] Registered new tserver with Master: df499f2ee4dd448c8edc34aad9d813f4 (127.0.192.193:33635)
I20260621 07:47:22.302986 3190 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.193@KRBTEST.COM'} at 127.0.192.193:35679
WARNING: no policy specified for kudu/127.0.192.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab.
Entry for principal kudu/127.0.192.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab.
I20260621 07:47:22.322665 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.194:0
--local_ip_for_outbound_sockets=127.0.192.194
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:35445
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:40501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:22.422176 3386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:22.422366 3386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:22.422432 3386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:22.426245 3386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:22.426291 3386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:22.426393 3386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.194
I20260621 07:47:22.431030 3386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:40501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:35445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.194
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:22.431871 3386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:22.432824 3386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:22.441212 3391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:22.441216 3392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:22.441356 3386 server_base.cc:1034] running on GCE node
W20260621 07:47:22.441288 3394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:22.441741 3386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:22.442199 3386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:22.443351 3386 hybrid_clock.cc:648] HybridClock initialized: now 1782028042443321 us; error 43 us; skew 500 ppm
Jun 21 07:47:22 dist-test-slave-lzx3 krb5kdc[3145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028042, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:22.445905 3386 init.cc:377] Logged in from keytab as kudu/127.0.192.194@KRBTEST.COM (short username kudu)
I20260621 07:47:22.446592 3386 webserver.cc:460] Webserver started at http://127.0.192.194:46321/ using document root <none> and password file <none>
I20260621 07:47:22.447055 3386 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:22.447141 3386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:22.447372 3386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:22.449251 3386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-1/data/instance:
uuid: "06a27a031f074eacb05045b24336ef57"
format_stamp: "Formatted at 2026-06-21 07:47:22 on dist-test-slave-lzx3"
server_key: "c00e4fd230c3d78efe1b96db96bb3d7a"
server_key_iv: "f9965088088482a80c264fdeb5ddabf7"
server_key_version: "encryptionkey@0"
I20260621 07:47:22.449749 3386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-1/wal/instance:
uuid: "06a27a031f074eacb05045b24336ef57"
format_stamp: "Formatted at 2026-06-21 07:47:22 on dist-test-slave-lzx3"
server_key: "c00e4fd230c3d78efe1b96db96bb3d7a"
server_key_iv: "f9965088088482a80c264fdeb5ddabf7"
server_key_version: "encryptionkey@0"
I20260621 07:47:22.452912 3386 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:47:22.455053 3401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:22.456053 3386 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:22.456179 3386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-1/wal
uuid: "06a27a031f074eacb05045b24336ef57"
format_stamp: "Formatted at 2026-06-21 07:47:22 on dist-test-slave-lzx3"
server_key: "c00e4fd230c3d78efe1b96db96bb3d7a"
server_key_iv: "f9965088088482a80c264fdeb5ddabf7"
server_key_version: "encryptionkey@0"
I20260621 07:47:22.456281 3386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:22.474294 3386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:22.477301 3386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:22.477521 3386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:22.478060 3386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:47:22.478904 3386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:47:22.478981 3386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:22.479056 3386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:47:22.479090 3386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:22.487236 3386 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.194:36505
I20260621 07:47:22.487329 3514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.194:36505 every 8 connection(s)
I20260621 07:47:22.488020 3386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
Jun 21 07:47:22 dist-test-slave-lzx3 krb5kdc[3145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028042, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:47:22.497650 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 3386
I20260621 07:47:22.497802 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-1/wal/instance
I20260621 07:47:22.498112 771 external_mini_cluster.cc:1428] Setting key ea2465f81ae9fda4d431bcf1bc911750
I20260621 07:47:22.500313 3229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:22.489662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:43597 (local address 127.0.192.254:35445)
0621 07:47:22.489800 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:22.489805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:22.490607 (+ 802us) server_negotiation.cc:407] Connection header received
0621 07:47:22.491742 (+ 1135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:22.491744 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:22.491786 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:22.491854 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:22.492853 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:22.493366 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:22.494396 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:22.494594 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:22.494666 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:22.496800 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:22.496803 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:22.496804 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:22.496824 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:22.498309 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:22.498776 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:22.498778 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:22.498779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:22.498891 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:22.499242 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:22.499245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:22.499246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:22.499512 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:22.499611 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:22.500111 (+ 500us) server_negotiation.cc:299] Negotiation successful
0621 07:47:22.500208 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 07:47:22.501374 3517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:22.490058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:35445 (local address 127.0.192.194:43597)
0621 07:47:22.490441 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:22.490503 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:22.491570 (+ 1067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:22.491994 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:22.492000 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:22.492277 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:22.492711 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:22.492740 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:22.493469 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:22.493471 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:22.494290 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:22.494298 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:22.494713 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:22.494715 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:22.494840 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:22.495374 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:22.495387 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:22.496673 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:22.498407 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:22.498411 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:22.498421 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:22.498650 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:22.498990 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:22.498992 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:22.498994 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:22.499119 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:22.499614 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:22.499619 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:22.499893 (+ 274us) client_negotiation.cc:769] Sending connection context
0621 07:47:22.500072 (+ 179us) client_negotiation.cc:240] Negotiation successful
0621 07:47:22.500398 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":108,"threads_started":1}
I20260621 07:47:22.502511 3515 heartbeater.cc:347] Connected to a master server at 127.0.192.254:35445
I20260621 07:47:22.502719 3515 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:22.503185 3515 heartbeater.cc:511] Master 127.0.192.254:35445 requested a full tablet report, sending...
I20260621 07:47:22.504258 3190 ts_manager.cc:194] Registered new tserver with Master: 06a27a031f074eacb05045b24336ef57 (127.0.192.194:36505)
I20260621 07:47:22.505153 3190 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.194@KRBTEST.COM'} at 127.0.192.194:43597
WARNING: no policy specified for kudu/127.0.192.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab.
Entry for principal kudu/127.0.192.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab.
I20260621 07:47:22.523249 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.195:0
--local_ip_for_outbound_sockets=127.0.192.195
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:35445
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:40501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:22.629384 3520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:22.629601 3520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:22.629638 3520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:22.633284 3520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:22.633337 3520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:22.633430 3520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.195
I20260621 07:47:22.638510 3520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:40501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:35445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.195
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:22.639364 3520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:22.640271 3520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:22.649482 3528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:22.649494 3526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:22.649482 3525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:22.649755 3520 server_base.cc:1034] running on GCE node
I20260621 07:47:22.650166 3520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:22.650784 3520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:22.651988 3520 hybrid_clock.cc:648] HybridClock initialized: now 1782028042651966 us; error 51 us; skew 500 ppm
Jun 21 07:47:22 dist-test-slave-lzx3 krb5kdc[3145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028042, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:22.654976 3520 init.cc:377] Logged in from keytab as kudu/127.0.192.195@KRBTEST.COM (short username kudu)
I20260621 07:47:22.655771 3520 webserver.cc:460] Webserver started at http://127.0.192.195:39641/ using document root <none> and password file <none>
I20260621 07:47:22.656307 3520 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:22.656405 3520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:22.656713 3520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:22.658815 3520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-2/data/instance:
uuid: "b7fd2a49c7ad40c395caf3b4f4f2e870"
format_stamp: "Formatted at 2026-06-21 07:47:22 on dist-test-slave-lzx3"
server_key: "f2d486a8c2bb21053372272d2c29c0cb"
server_key_iv: "34b9fb0a7631e4508f75b61ddee05e16"
server_key_version: "encryptionkey@0"
I20260621 07:47:22.659400 3520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-2/wal/instance:
uuid: "b7fd2a49c7ad40c395caf3b4f4f2e870"
format_stamp: "Formatted at 2026-06-21 07:47:22 on dist-test-slave-lzx3"
server_key: "f2d486a8c2bb21053372272d2c29c0cb"
server_key_iv: "34b9fb0a7631e4508f75b61ddee05e16"
server_key_version: "encryptionkey@0"
I20260621 07:47:22.663249 3520 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 07:47:22.665627 3535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:22.666798 3520 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 07:47:22.666930 3520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-2/wal
uuid: "b7fd2a49c7ad40c395caf3b4f4f2e870"
format_stamp: "Formatted at 2026-06-21 07:47:22 on dist-test-slave-lzx3"
server_key: "f2d486a8c2bb21053372272d2c29c0cb"
server_key_iv: "34b9fb0a7631e4508f75b61ddee05e16"
server_key_version: "encryptionkey@0"
I20260621 07:47:22.667040 3520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:22.680081 3520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:22.682901 3520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:22.683086 3520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:22.683576 3520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:47:22.684319 3520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:47:22.684396 3520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:22.684455 3520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:47:22.684515 3520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:22.692379 3520 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.195:39519
I20260621 07:47:22.692404 3648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.195:39519 every 8 connection(s)
I20260621 07:47:22.693243 3520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
I20260621 07:47:22.698665 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 3520
I20260621 07:47:22.698766 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-2/wal/instance
I20260621 07:47:22.699036 771 external_mini_cluster.cc:1428] Setting key d8feac82e8910b2f19580d070603eae1
Jun 21 07:47:22 dist-test-slave-lzx3 krb5kdc[3145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028042, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:47:22.706035 3229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:22.694883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:35359 (local address 127.0.192.254:35445)
0621 07:47:22.695030 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:22.695034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:22.695655 (+ 621us) server_negotiation.cc:407] Connection header received
0621 07:47:22.696995 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:22.696998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:22.697055 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:22.697132 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:22.698165 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:22.698819 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:22.700099 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:22.700350 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:22.700468 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:22.702757 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:22.702759 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:22.702761 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:22.702784 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:22.704175 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:22.704689 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:22.704692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:22.704693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:22.704775 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:22.705143 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:22.705145 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:22.705146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:22.705309 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:22.705387 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:22.705842 (+ 455us) server_negotiation.cc:299] Negotiation successful
0621 07:47:22.705928 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260621 07:47:22.706634 3651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:22.695135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:35445 (local address 127.0.192.195:35359)
0621 07:47:22.695511 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:22.695547 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:22.696788 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:22.697300 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:22.697306 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:22.697599 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:22.698010 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:22.698021 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:22.698948 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:22.698951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:22.699972 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:22.699983 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:22.700502 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:22.700506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:22.700676 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:22.701185 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:22.701199 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:22.702602 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:22.704291 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:22.704295 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:22.704304 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:22.704542 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:22.704889 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:22.704892 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:22.704893 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:22.705040 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:22.705400 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:22.705404 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:22.705589 (+ 185us) client_negotiation.cc:769] Sending connection context
0621 07:47:22.705729 (+ 140us) client_negotiation.cc:240] Negotiation successful
0621 07:47:22.705938 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":99,"threads_started":1}
I20260621 07:47:22.707644 3649 heartbeater.cc:347] Connected to a master server at 127.0.192.254:35445
I20260621 07:47:22.707844 3649 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:22.708271 3649 heartbeater.cc:511] Master 127.0.192.254:35445 requested a full tablet report, sending...
I20260621 07:47:22.709215 3190 ts_manager.cc:194] Registered new tserver with Master: b7fd2a49c7ad40c395caf3b4f4f2e870 (127.0.192.195:39519)
I20260621 07:47:22.709722 3190 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.195@KRBTEST.COM'} at 127.0.192.195:35359
I20260621 07:47:22.713447 771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260621 07:47:22.725150 3229 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:22.723866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52792 (local address 127.0.192.254:35445)
0621 07:47:22.723986 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:22.723990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:22.724161 (+ 171us) server_negotiation.cc:407] Connection header received
0621 07:47:22.724243 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:22.724244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:22.724279 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:22.724344 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:22.725021 (+ 677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52792: BlockingRecv error: recv got EOF from 127.0.0.1:52792 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 07:47:22.726092 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 3252
I20260621 07:47:22.731106 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 3386
I20260621 07:47:22.736112 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 3520
I20260621 07:47:22.741086 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 3158
2026-06-21T07:47:22Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1953 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028028522580-771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:47:22 dist-test-slave-lzx3 krb5kdc[3661](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:47:22 dist-test-slave-lzx3 krb5kdc[3661](info): set up 2 sockets
Jun 21 07:47:22 dist-test-slave-lzx3 krb5kdc[3661](info): commencing operation
krb5kdc: starting...
W20260621 07:47:23.724310 771 mini_kdc.cc:121] Time spent starting KDC: real 0.962s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028028522580-771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028028522580-771-0/krb5kdc/test-user.keytab.
Jun 21 07:47:23 dist-test-slave-lzx3 krb5kdc[3661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:47:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:47:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.192.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
I20260621 07:47:23.835944 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:36617
--webserver_interface=127.0.192.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:41475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:36617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:23.937110 3674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:23.937337 3674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:23.937403 3674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:23.940986 3674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:47:23.941052 3674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:23.941084 3674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:23.941161 3674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:47:23.941197 3674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:47:23.945784 3674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:41475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:36617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:36617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:23.946611 3674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:23.947458 3674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:23.954924 3682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:23.954941 3680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:23.955066 3679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:23.955431 3674 server_base.cc:1034] running on GCE node
I20260621 07:47:23.955858 3674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:23.956864 3674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:23.958057 3674 hybrid_clock.cc:648] HybridClock initialized: now 1782028043958034 us; error 47 us; skew 500 ppm
Jun 21 07:47:23 dist-test-slave-lzx3 krb5kdc[3661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028043, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:23.960500 3674 init.cc:377] Logged in from keytab as kudu/127.0.192.254@KRBTEST.COM (short username kudu)
I20260621 07:47:23.961422 3674 webserver.cc:460] Webserver started at http://127.0.192.254:44863/ using document root <none> and password file <none>
I20260621 07:47:23.961887 3674 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:23.961941 3674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:23.962088 3674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:23.963940 3674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "50b35f5f4212449b87c8454fbdc0f82f"
format_stamp: "Formatted at 2026-06-21 07:47:23 on dist-test-slave-lzx3"
server_key: "e125f36872f811bf288e3cb284ba2c33"
server_key_iv: "c915e76d502fa7c25f426f964116bbc2"
server_key_version: "encryptionkey@0"
I20260621 07:47:23.964426 3674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "50b35f5f4212449b87c8454fbdc0f82f"
format_stamp: "Formatted at 2026-06-21 07:47:23 on dist-test-slave-lzx3"
server_key: "e125f36872f811bf288e3cb284ba2c33"
server_key_iv: "c915e76d502fa7c25f426f964116bbc2"
server_key_version: "encryptionkey@0"
I20260621 07:47:23.967954 3674 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260621 07:47:23.970157 3689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:23.971259 3674 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 07:47:23.971347 3674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "50b35f5f4212449b87c8454fbdc0f82f"
format_stamp: "Formatted at 2026-06-21 07:47:23 on dist-test-slave-lzx3"
server_key: "e125f36872f811bf288e3cb284ba2c33"
server_key_iv: "c915e76d502fa7c25f426f964116bbc2"
server_key_version: "encryptionkey@0"
I20260621 07:47:23.971419 3674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:23.984804 3674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:23.987066 3674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:23.987182 3674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:23.993414 3674 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:36617
I20260621 07:47:23.993703 3741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:36617 every 8 connection(s)
I20260621 07:47:23.994282 3674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:47:23.996852 3742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:24.000763 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 3674
I20260621 07:47:24.000874 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:47:24.001148 771 external_mini_cluster.cc:1428] Setting key cb0fd94258d23b9502a41698ae900619
I20260621 07:47:24.002316 3742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50b35f5f4212449b87c8454fbdc0f82f: Bootstrap starting.
I20260621 07:47:24.004766 3742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50b35f5f4212449b87c8454fbdc0f82f: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:24.005497 3742 log.cc:824] T 00000000000000000000000000000000 P 50b35f5f4212449b87c8454fbdc0f82f: Log is configured to *not* fsync() on all Append() calls
I20260621 07:47:24.007418 3742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50b35f5f4212449b87c8454fbdc0f82f: No bootstrap required, opened a new log
Jun 21 07:47:24 dist-test-slave-lzx3 krb5kdc[3661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:47:24.010877 3742 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50b35f5f4212449b87c8454fbdc0f82f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50b35f5f4212449b87c8454fbdc0f82f" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 36617 } }
I20260621 07:47:24.011107 3742 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50b35f5f4212449b87c8454fbdc0f82f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:24.011190 3742 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50b35f5f4212449b87c8454fbdc0f82f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50b35f5f4212449b87c8454fbdc0f82f, State: Initialized, Role: FOLLOWER
I20260621 07:47:24.011569 3742 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50b35f5f4212449b87c8454fbdc0f82f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50b35f5f4212449b87c8454fbdc0f82f" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 36617 } }
I20260621 07:47:24.011714 3742 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50b35f5f4212449b87c8454fbdc0f82f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:47:24.011776 3742 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50b35f5f4212449b87c8454fbdc0f82f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:47:24.011888 3742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50b35f5f4212449b87c8454fbdc0f82f [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:24.012705 3742 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50b35f5f4212449b87c8454fbdc0f82f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50b35f5f4212449b87c8454fbdc0f82f" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 36617 } }
I20260621 07:47:24.013063 3742 leader_election.cc:302] T 00000000000000000000000000000000 P 50b35f5f4212449b87c8454fbdc0f82f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50b35f5f4212449b87c8454fbdc0f82f; no voters:
I20260621 07:47:24.013383 3742 leader_election.cc:288] T 00000000000000000000000000000000 P 50b35f5f4212449b87c8454fbdc0f82f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:47:24.013443 3747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50b35f5f4212449b87c8454fbdc0f82f [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:47:24.013698 3747 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50b35f5f4212449b87c8454fbdc0f82f [term 1 LEADER]: Becoming Leader. State: Replica: 50b35f5f4212449b87c8454fbdc0f82f, State: Running, Role: LEADER
I20260621 07:47:24.014024 3747 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50b35f5f4212449b87c8454fbdc0f82f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50b35f5f4212449b87c8454fbdc0f82f" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 36617 } }
I20260621 07:47:24.014513 3745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:24.002273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47744 (local address 127.0.192.254:36617)
0621 07:47:24.002642 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:24.002657 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:24.002690 (+ 33us) server_negotiation.cc:407] Connection header received
0621 07:47:24.003326 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:24.003342 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:24.003641 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:24.003897 (+ 256us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:24.004947 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:24.005938 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:24.007031 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:24.007404 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:24.007541 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:24.009896 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:24.009900 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:24.009911 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:24.009932 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:24.011632 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:24.012064 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:24.012070 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:24.012075 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:24.012189 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:24.012469 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:24.012474 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:24.012475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:24.012957 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:24.013161 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:24.013386 (+ 225us) server_negotiation.cc:299] Negotiation successful
0621 07:47:24.013671 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":106,"threads_started":1}
I20260621 07:47:24.014611 3742 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50b35f5f4212449b87c8454fbdc0f82f [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:47:24.015115 3747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50b35f5f4212449b87c8454fbdc0f82f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50b35f5f4212449b87c8454fbdc0f82f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50b35f5f4212449b87c8454fbdc0f82f" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 36617 } } }
I20260621 07:47:24.015313 3747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50b35f5f4212449b87c8454fbdc0f82f [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:24.015452 3748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50b35f5f4212449b87c8454fbdc0f82f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50b35f5f4212449b87c8454fbdc0f82f. Latest consensus state: current_term: 1 leader_uuid: "50b35f5f4212449b87c8454fbdc0f82f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50b35f5f4212449b87c8454fbdc0f82f" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 36617 } } }
I20260621 07:47:24.015689 3748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50b35f5f4212449b87c8454fbdc0f82f [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:24.016815 3755 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:47:24.017380 771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:47:24.019524 3755 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:47:24.025076 3755 catalog_manager.cc:1347] Generated new cluster ID: 72a662fb0ae048db8b61d0b7a1a3b7a5
I20260621 07:47:24.025182 3755 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:47:24.033344 3745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:24.028831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47756 (local address 127.0.192.254:36617)
0621 07:47:24.028935 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:24.028939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:24.029172 (+ 233us) server_negotiation.cc:407] Connection header received
0621 07:47:24.029313 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:24.029316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:24.029359 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:24.029440 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:24.029905 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:24.030435 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:24.031230 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:24.031454 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:24.031555 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:24.032336 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:24.032340 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:24.032342 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:47:24.032822 (+ 480us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:24.032869 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:47:24.032887 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:24.032992 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:24.033165 (+ 173us) server_negotiation.cc:299] Negotiation successful
0621 07:47:24.033215 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260621 07:47:24.034160 3755 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:47:24.035293 3755 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:47:24.045531 3755 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50b35f5f4212449b87c8454fbdc0f82f: Generated new TSK 0
I20260621 07:47:24.046155 3755 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:47:24.052240 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 3674
2026-06-21T07:47:24Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1309 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:47:24 dist-test-slave-lzx3 krb5kdc[3775](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:47:24 dist-test-slave-lzx3 krb5kdc[3775](info): set up 2 sockets
Jun 21 07:47:24 dist-test-slave-lzx3 krb5kdc[3775](info): commencing operation
krb5kdc: starting...
W20260621 07:47:25.028291 771 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/krb5kdc/test-user.keytab.
Jun 21 07:47:25 dist-test-slave-lzx3 krb5kdc[3775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:47:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:47:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.192.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
I20260621 07:47:25.135663 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:37405
--webserver_interface=127.0.192.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:34705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:37405
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:25.236765 3788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:25.237012 3788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:25.237072 3788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:25.241421 3788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:47:25.241498 3788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:25.241532 3788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:25.241565 3788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:47:25.241596 3788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:47:25.246798 3788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:34705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:37405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:37405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:25.247725 3788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:25.248680 3788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:25.256145 3794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:25.256151 3793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:25.256218 3796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:25.256458 3788 server_base.cc:1034] running on GCE node
I20260621 07:47:25.257091 3788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:25.258193 3788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:25.259362 3788 hybrid_clock.cc:648] HybridClock initialized: now 1782028045259345 us; error 34 us; skew 500 ppm
Jun 21 07:47:25 dist-test-slave-lzx3 krb5kdc[3775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:25.262164 3788 init.cc:377] Logged in from keytab as kudu/127.0.192.254@KRBTEST.COM (short username kudu)
I20260621 07:47:25.263025 3788 webserver.cc:460] Webserver started at http://127.0.192.254:33569/ using document root <none> and password file <none>
I20260621 07:47:25.263593 3788 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:25.263697 3788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:25.263947 3788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:25.265973 3788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "9574002af70849098a1625106d6bbd05"
format_stamp: "Formatted at 2026-06-21 07:47:25 on dist-test-slave-lzx3"
server_key: "9f102628004e2549b1dbc848207de34d"
server_key_iv: "0c0727a7de0d83baea24b7db9ac1ae04"
server_key_version: "encryptionkey@0"
I20260621 07:47:25.266556 3788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "9574002af70849098a1625106d6bbd05"
format_stamp: "Formatted at 2026-06-21 07:47:25 on dist-test-slave-lzx3"
server_key: "9f102628004e2549b1dbc848207de34d"
server_key_iv: "0c0727a7de0d83baea24b7db9ac1ae04"
server_key_version: "encryptionkey@0"
I20260621 07:47:25.269964 3788 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 07:47:25.272083 3803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:25.273281 3788 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:25.273433 3788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "9574002af70849098a1625106d6bbd05"
format_stamp: "Formatted at 2026-06-21 07:47:25 on dist-test-slave-lzx3"
server_key: "9f102628004e2549b1dbc848207de34d"
server_key_iv: "0c0727a7de0d83baea24b7db9ac1ae04"
server_key_version: "encryptionkey@0"
I20260621 07:47:25.273548 3788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:25.298997 3788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:25.302013 3788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:25.302208 3788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:25.308609 3788 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:37405
I20260621 07:47:25.308825 3855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:37405 every 8 connection(s)
I20260621 07:47:25.309468 3788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:47:25.310454 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 3788
I20260621 07:47:25.310562 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:47:25.310812 771 external_mini_cluster.cc:1428] Setting key b53a0c022a640f639bf1e2620a57c967
I20260621 07:47:25.312333 3856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:25.318325 3856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9574002af70849098a1625106d6bbd05: Bootstrap starting.
Jun 21 07:47:25 dist-test-slave-lzx3 krb5kdc[3775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:47:25.320626 3856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9574002af70849098a1625106d6bbd05: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:25.321412 3856 log.cc:824] T 00000000000000000000000000000000 P 9574002af70849098a1625106d6bbd05: Log is configured to *not* fsync() on all Append() calls
I20260621 07:47:25.323299 3856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9574002af70849098a1625106d6bbd05: No bootstrap required, opened a new log
I20260621 07:47:25.324672 3859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:25.311956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48900 (local address 127.0.192.254:37405)
0621 07:47:25.312313 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:25.312330 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:25.312368 (+ 38us) server_negotiation.cc:407] Connection header received
0621 07:47:25.313150 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:25.313172 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:25.313560 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:25.313911 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:25.314527 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:25.315928 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:25.316706 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:25.317337 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:25.317486 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:25.319847 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:25.319852 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:25.319864 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:25.319889 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:25.321950 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:25.322367 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:25.322375 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:25.322381 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:25.322515 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:25.322786 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:25.322791 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:25.322795 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:25.323257 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:25.323468 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:25.323733 (+ 265us) server_negotiation.cc:299] Negotiation successful
0621 07:47:25.323980 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":103,"threads_started":1}
I20260621 07:47:25.326709 3856 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9574002af70849098a1625106d6bbd05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9574002af70849098a1625106d6bbd05" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 37405 } }
I20260621 07:47:25.326934 3856 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9574002af70849098a1625106d6bbd05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:25.327008 3856 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9574002af70849098a1625106d6bbd05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9574002af70849098a1625106d6bbd05, State: Initialized, Role: FOLLOWER
I20260621 07:47:25.327348 3856 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9574002af70849098a1625106d6bbd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9574002af70849098a1625106d6bbd05" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 37405 } }
I20260621 07:47:25.327481 3856 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9574002af70849098a1625106d6bbd05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:47:25.327560 3856 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9574002af70849098a1625106d6bbd05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:47:25.327642 3856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9574002af70849098a1625106d6bbd05 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:25.328402 3856 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9574002af70849098a1625106d6bbd05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9574002af70849098a1625106d6bbd05" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 37405 } }
I20260621 07:47:25.328727 3856 leader_election.cc:302] T 00000000000000000000000000000000 P 9574002af70849098a1625106d6bbd05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9574002af70849098a1625106d6bbd05; no voters:
I20260621 07:47:25.329001 3856 leader_election.cc:288] T 00000000000000000000000000000000 P 9574002af70849098a1625106d6bbd05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:47:25.329120 3861 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9574002af70849098a1625106d6bbd05 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:47:25.329348 3861 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9574002af70849098a1625106d6bbd05 [term 1 LEADER]: Becoming Leader. State: Replica: 9574002af70849098a1625106d6bbd05, State: Running, Role: LEADER
I20260621 07:47:25.329679 3861 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9574002af70849098a1625106d6bbd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9574002af70849098a1625106d6bbd05" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 37405 } }
I20260621 07:47:25.330089 3856 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9574002af70849098a1625106d6bbd05 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:47:25.331096 3863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9574002af70849098a1625106d6bbd05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9574002af70849098a1625106d6bbd05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9574002af70849098a1625106d6bbd05" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 37405 } } }
I20260621 07:47:25.331223 3863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9574002af70849098a1625106d6bbd05 [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:25.331519 3862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9574002af70849098a1625106d6bbd05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9574002af70849098a1625106d6bbd05. Latest consensus state: current_term: 1 leader_uuid: "9574002af70849098a1625106d6bbd05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9574002af70849098a1625106d6bbd05" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 37405 } } }
I20260621 07:47:25.331632 3870 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:47:25.333117 3862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9574002af70849098a1625106d6bbd05 [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:25.334650 3870 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:47:25.339509 3870 catalog_manager.cc:1347] Generated new cluster ID: 319834c7ef3b44789d269a58d4861ca9
I20260621 07:47:25.339577 3870 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:47:25.344866 3870 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:47:25.345677 3870 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:47:25.351751 3870 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9574002af70849098a1625106d6bbd05: Generated new TSK 0
I20260621 07:47:25.352300 3870 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.192.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab.
Entry for principal kudu/127.0.192.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab.
I20260621 07:47:25.380740 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.193:0
--local_ip_for_outbound_sockets=127.0.192.193
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:37405
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:34705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:25.480172 3882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:25.480400 3882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:25.480461 3882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:25.484282 3882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:25.484333 3882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:25.484448 3882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.193
I20260621 07:47:25.489151 3882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:34705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:37405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.193
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:25.489985 3882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:25.490969 3882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 07:47:25.499545 3882 server_base.cc:1034] running on GCE node
W20260621 07:47:25.499504 3887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:25.499493 3888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:25.499495 3890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:25.500061 3882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:25.500753 3882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:25.501932 3882 hybrid_clock.cc:648] HybridClock initialized: now 1782028045501917 us; error 27 us; skew 500 ppm
Jun 21 07:47:25 dist-test-slave-lzx3 krb5kdc[3775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:25.504350 3882 init.cc:377] Logged in from keytab as kudu/127.0.192.193@KRBTEST.COM (short username kudu)
I20260621 07:47:25.505198 3882 webserver.cc:460] Webserver started at http://127.0.192.193:40585/ using document root <none> and password file <none>
I20260621 07:47:25.505745 3882 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:25.505832 3882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:25.506032 3882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:25.507639 3882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-0/data/instance:
uuid: "9457ffae826d428eb615f4846d921fc7"
format_stamp: "Formatted at 2026-06-21 07:47:25 on dist-test-slave-lzx3"
server_key: "a0cf801a2776a8d114cfbbc1cf0017ee"
server_key_iv: "4722983caadbad0866e49fdfe86f8d76"
server_key_version: "encryptionkey@0"
I20260621 07:47:25.508106 3882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-0/wal/instance:
uuid: "9457ffae826d428eb615f4846d921fc7"
format_stamp: "Formatted at 2026-06-21 07:47:25 on dist-test-slave-lzx3"
server_key: "a0cf801a2776a8d114cfbbc1cf0017ee"
server_key_iv: "4722983caadbad0866e49fdfe86f8d76"
server_key_version: "encryptionkey@0"
I20260621 07:47:25.511209 3882 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260621 07:47:25.513160 3897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:25.514171 3882 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:25.514329 3882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-0/wal
uuid: "9457ffae826d428eb615f4846d921fc7"
format_stamp: "Formatted at 2026-06-21 07:47:25 on dist-test-slave-lzx3"
server_key: "a0cf801a2776a8d114cfbbc1cf0017ee"
server_key_iv: "4722983caadbad0866e49fdfe86f8d76"
server_key_version: "encryptionkey@0"
I20260621 07:47:25.514441 3882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:25.526600 3882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:25.529342 3882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:25.529520 3882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:25.529964 3882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:47:25.530673 3882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:47:25.530740 3882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:25.530799 3882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:47:25.530840 3882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:25.539312 3882 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.193:40367
I20260621 07:47:25.539353 4010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.193:40367 every 8 connection(s)
I20260621 07:47:25.540062 3882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
I20260621 07:47:25.545711 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 3882
I20260621 07:47:25.545801 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-0/wal/instance
I20260621 07:47:25.546097 771 external_mini_cluster.cc:1428] Setting key 8ae5aa300d5c82fb3ee591ebe52a3dc4
Jun 21 07:47:25 dist-test-slave-lzx3 krb5kdc[3775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:47:25.552918 3859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:25.541601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:45005 (local address 127.0.192.254:37405)
0621 07:47:25.541737 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:25.541741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:25.542443 (+ 702us) server_negotiation.cc:407] Connection header received
0621 07:47:25.543632 (+ 1189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:25.543634 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:25.543686 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:25.543790 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:25.544881 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:25.545409 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:25.546732 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:25.546949 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:25.547056 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:25.549424 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:25.549429 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:25.549432 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:25.549457 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:25.550947 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:25.551470 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:25.551476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:25.551477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:25.551556 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:25.551849 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:25.551854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:25.551859 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:25.552082 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:25.552188 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:25.552584 (+ 396us) server_negotiation.cc:299] Negotiation successful
0621 07:47:25.552720 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 07:47:25.553507 4013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:25.541901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:37405 (local address 127.0.192.193:45005)
0621 07:47:25.542309 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:25.542342 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:25.543433 (+ 1091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:25.543944 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:25.543949 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:25.544231 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:25.544727 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:25.544741 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:25.545559 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:25.545562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:25.546622 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:25.546631 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:25.547082 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:25.547086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:25.547259 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:25.547825 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:25.547839 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:25.549274 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:25.551053 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:25.551057 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:25.551066 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:25.551369 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:25.551646 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:25.551647 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:25.551649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:25.551757 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:25.552181 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:25.552185 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:25.552355 (+ 170us) client_negotiation.cc:769] Sending connection context
0621 07:47:25.552486 (+ 131us) client_negotiation.cc:240] Negotiation successful
0621 07:47:25.552744 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":110,"threads_started":1}
I20260621 07:47:25.554708 4011 heartbeater.cc:347] Connected to a master server at 127.0.192.254:37405
I20260621 07:47:25.554931 4011 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:25.555356 4011 heartbeater.cc:511] Master 127.0.192.254:37405 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.192.194@KRBTEST.COM; defaulting to no policy
I20260621 07:47:25.556799 3820 ts_manager.cc:194] Registered new tserver with Master: 9457ffae826d428eb615f4846d921fc7 (127.0.192.193:40367)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.194@KRBTEST.COM" created.
I20260621 07:47:25.557878 3820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.193@KRBTEST.COM'} at 127.0.192.193:45005
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab.
Entry for principal kudu/127.0.192.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab.
I20260621 07:47:25.575003 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.194:0
--local_ip_for_outbound_sockets=127.0.192.194
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:37405
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:34705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:25.682003 4016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:25.682266 4016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:25.682343 4016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:25.686751 4016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:25.686831 4016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:25.686926 4016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.194
I20260621 07:47:25.691737 4016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:34705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:37405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.194
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:25.692670 4016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:25.693629 4016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:25.702930 4022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:25.702955 4021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:25.702973 4024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:25.703162 4016 server_base.cc:1034] running on GCE node
I20260621 07:47:25.703555 4016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:25.704113 4016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:25.705289 4016 hybrid_clock.cc:648] HybridClock initialized: now 1782028045705275 us; error 26 us; skew 500 ppm
Jun 21 07:47:25 dist-test-slave-lzx3 krb5kdc[3775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:25.707783 4016 init.cc:377] Logged in from keytab as kudu/127.0.192.194@KRBTEST.COM (short username kudu)
I20260621 07:47:25.708484 4016 webserver.cc:460] Webserver started at http://127.0.192.194:33839/ using document root <none> and password file <none>
I20260621 07:47:25.708973 4016 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:25.709026 4016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:25.709230 4016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:25.711088 4016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-1/data/instance:
uuid: "4b857201a7c24220b34a289a47e2aade"
format_stamp: "Formatted at 2026-06-21 07:47:25 on dist-test-slave-lzx3"
server_key: "de6fc2d0a36e1ae3befb6bd134c19824"
server_key_iv: "6c6810ca556be0b8e4ab187b16cdc57d"
server_key_version: "encryptionkey@0"
I20260621 07:47:25.711670 4016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-1/wal/instance:
uuid: "4b857201a7c24220b34a289a47e2aade"
format_stamp: "Formatted at 2026-06-21 07:47:25 on dist-test-slave-lzx3"
server_key: "de6fc2d0a36e1ae3befb6bd134c19824"
server_key_iv: "6c6810ca556be0b8e4ab187b16cdc57d"
server_key_version: "encryptionkey@0"
I20260621 07:47:25.715091 4016 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260621 07:47:25.717269 4031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:25.718392 4016 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:25.718495 4016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-1/wal
uuid: "4b857201a7c24220b34a289a47e2aade"
format_stamp: "Formatted at 2026-06-21 07:47:25 on dist-test-slave-lzx3"
server_key: "de6fc2d0a36e1ae3befb6bd134c19824"
server_key_iv: "6c6810ca556be0b8e4ab187b16cdc57d"
server_key_version: "encryptionkey@0"
I20260621 07:47:25.718585 4016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:25.741595 4016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:25.744493 4016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:25.744694 4016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:25.745220 4016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:47:25.745983 4016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:47:25.746049 4016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:25.746115 4016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:47:25.746150 4016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:25.754805 4016 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.194:41763
I20260621 07:47:25.754829 4144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.194:41763 every 8 connection(s)
I20260621 07:47:25.755558 4016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
I20260621 07:47:25.760782 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 4016
I20260621 07:47:25.760907 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-1/wal/instance
I20260621 07:47:25.761176 771 external_mini_cluster.cc:1428] Setting key f445e8fa894430c994d141fb1eebb20e
Jun 21 07:47:25 dist-test-slave-lzx3 krb5kdc[3775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:47:25.769282 3859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:25.757166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:34259 (local address 127.0.192.254:37405)
0621 07:47:25.757322 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:25.757327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:25.758128 (+ 801us) server_negotiation.cc:407] Connection header received
0621 07:47:25.759337 (+ 1209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:25.759339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:25.759385 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:25.759456 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:25.760464 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:25.761266 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:25.762811 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:25.763176 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:25.763296 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:25.765780 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:25.765784 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:25.765789 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:25.765816 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:25.767243 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:25.767730 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:25.767735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:25.767737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:25.767814 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:25.768150 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:25.768153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:25.768155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:25.768320 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:25.768426 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:25.769008 (+ 582us) server_negotiation.cc:299] Negotiation successful
0621 07:47:25.769121 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 07:47:25.770090 4147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:25.757571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:37405 (local address 127.0.192.194:34259)
0621 07:47:25.757983 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:25.758020 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:25.759157 (+ 1137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:25.759598 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:25.759604 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:25.759924 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:25.760342 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:25.760354 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:25.761422 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:25.761425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:25.762688 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:25.762697 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:25.763308 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:25.763311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:25.763446 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:25.763981 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:25.763996 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:25.765628 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:25.767367 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:25.767373 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:25.767385 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:25.767627 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:25.767911 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:25.767915 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:25.767917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:25.768048 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:25.768422 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:25.768428 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:25.768744 (+ 316us) client_negotiation.cc:769] Sending connection context
0621 07:47:25.768953 (+ 209us) client_negotiation.cc:240] Negotiation successful
0621 07:47:25.769240 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.0.192.195@KRBTEST.COM; defaulting to no policy
I20260621 07:47:25.771140 4145 heartbeater.cc:347] Connected to a master server at 127.0.192.254:37405
I20260621 07:47:25.771402 4145 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:25.771876 4145 heartbeater.cc:511] Master 127.0.192.254:37405 requested a full tablet report, sending...
I20260621 07:47:25.772810 3820 ts_manager.cc:194] Registered new tserver with Master: 4b857201a7c24220b34a289a47e2aade (127.0.192.194:41763)
I20260621 07:47:25.773388 3820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.194@KRBTEST.COM'} at 127.0.192.194:34259
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab.
Entry for principal kudu/127.0.192.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab.
I20260621 07:47:25.788971 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.195:0
--local_ip_for_outbound_sockets=127.0.192.195
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:37405
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:34705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:25.893098 4150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:25.893359 4150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:25.893433 4150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:25.897771 4150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:25.897857 4150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:25.897943 4150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.195
I20260621 07:47:25.902679 4150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:34705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:37405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.195
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:25.903617 4150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:25.904495 4150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:25.913211 4156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:25.913194 4155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:25.913409 4150 server_base.cc:1034] running on GCE node
W20260621 07:47:25.913180 4158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:25.913859 4150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:25.914364 4150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:25.915521 4150 hybrid_clock.cc:648] HybridClock initialized: now 1782028045915493 us; error 38 us; skew 500 ppm
Jun 21 07:47:25 dist-test-slave-lzx3 krb5kdc[3775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:25.921173 4150 init.cc:377] Logged in from keytab as kudu/127.0.192.195@KRBTEST.COM (short username kudu)
I20260621 07:47:25.922001 4150 webserver.cc:460] Webserver started at http://127.0.192.195:33855/ using document root <none> and password file <none>
I20260621 07:47:25.922535 4150 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:25.922590 4150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:25.922801 4150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:25.924527 4150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-2/data/instance:
uuid: "0428fb60ecab471f8522a164a32e62e4"
format_stamp: "Formatted at 2026-06-21 07:47:25 on dist-test-slave-lzx3"
server_key: "f0b12ec63bb9458b81ab57143a27efca"
server_key_iv: "f5a9a9b5f26506b14bf9acb4e495ee70"
server_key_version: "encryptionkey@0"
I20260621 07:47:25.925169 4150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-2/wal/instance:
uuid: "0428fb60ecab471f8522a164a32e62e4"
format_stamp: "Formatted at 2026-06-21 07:47:25 on dist-test-slave-lzx3"
server_key: "f0b12ec63bb9458b81ab57143a27efca"
server_key_iv: "f5a9a9b5f26506b14bf9acb4e495ee70"
server_key_version: "encryptionkey@0"
I20260621 07:47:25.928536 4150 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:47:25.930878 4165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:25.932101 4150 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260621 07:47:25.932265 4150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-2/wal
uuid: "0428fb60ecab471f8522a164a32e62e4"
format_stamp: "Formatted at 2026-06-21 07:47:25 on dist-test-slave-lzx3"
server_key: "f0b12ec63bb9458b81ab57143a27efca"
server_key_iv: "f5a9a9b5f26506b14bf9acb4e495ee70"
server_key_version: "encryptionkey@0"
I20260621 07:47:25.932377 4150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:25.946026 4150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:25.948295 4150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:25.948460 4150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:25.948952 4150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:47:25.949601 4150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:47:25.949668 4150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:25.949726 4150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:47:25.949757 4150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:25.957276 4150 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.195:34563
I20260621 07:47:25.957391 4278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.195:34563 every 8 connection(s)
I20260621 07:47:25.958067 4150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
I20260621 07:47:25.963876 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 4150
I20260621 07:47:25.963972 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782028028522580-771-0/minicluster-data/ts-2/wal/instance
I20260621 07:47:25.964244 771 external_mini_cluster.cc:1428] Setting key da9b04ec11936fa1ab817d3e100dc5e0
Jun 21 07:47:25 dist-test-slave-lzx3 krb5kdc[3775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:47:25.970737 3859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:25.959586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:43297 (local address 127.0.192.254:37405)
0621 07:47:25.959736 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:25.959740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:25.960482 (+ 742us) server_negotiation.cc:407] Connection header received
0621 07:47:25.961810 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:25.961813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:25.961860 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:25.961935 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:25.962967 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:25.963483 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:25.964770 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:25.965084 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:25.965182 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:25.967609 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:25.967612 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:25.967614 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:25.967634 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:25.969124 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:25.969574 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:25.969577 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:25.969578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:25.969645 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:25.969922 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:25.969924 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:25.969925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:25.970078 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:25.970149 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:25.970549 (+ 400us) server_negotiation.cc:299] Negotiation successful
0621 07:47:25.970631 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260621 07:47:25.971483 4281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:25.959943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:37405 (local address 127.0.192.195:43297)
0621 07:47:25.960326 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:25.960368 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:25.961605 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:25.962087 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:25.962092 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:25.962406 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:25.962811 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:25.962822 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:25.963585 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:25.963587 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:25.964641 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:25.964652 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:25.965229 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:25.965238 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:25.965376 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:25.965977 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:25.965996 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:25.967471 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:25.969229 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:25.969233 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:25.969243 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:25.969484 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:25.969724 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:25.969726 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:25.969727 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:25.969844 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:25.970164 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:25.970168 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:25.970369 (+ 201us) client_negotiation.cc:769] Sending connection context
0621 07:47:25.970505 (+ 136us) client_negotiation.cc:240] Negotiation successful
0621 07:47:25.970816 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":98,"threads_started":1}
I20260621 07:47:25.972538 4279 heartbeater.cc:347] Connected to a master server at 127.0.192.254:37405
I20260621 07:47:25.972826 4279 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:25.973281 4279 heartbeater.cc:511] Master 127.0.192.254:37405 requested a full tablet report, sending...
I20260621 07:47:25.974210 3820 ts_manager.cc:194] Registered new tserver with Master: 0428fb60ecab471f8522a164a32e62e4 (127.0.192.195:34563)
I20260621 07:47:25.974763 3820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.195@KRBTEST.COM'} at 127.0.192.195:43297
I20260621 07:47:25.977983 771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 21 07:47:25 dist-test-slave-lzx3 krb5kdc[3775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028045, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 21 07:47:25 dist-test-slave-lzx3 krb5kdc[3775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028045, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:47:25.999817 3859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:25.992876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48912 (local address 127.0.192.254:37405)
0621 07:47:25.992977 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:25.992979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:25.993163 (+ 184us) server_negotiation.cc:407] Connection header received
0621 07:47:25.993249 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:25.993251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:25.993284 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:25.993346 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:25.993960 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:25.994425 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:25.995198 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:25.995388 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:25.995459 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:25.997584 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:25.997587 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:25.997588 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:25.997606 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:25.998640 (+ 1034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:25.998986 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:25.998988 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:25.998989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:25.999048 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:25.999247 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:25.999249 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:25.999250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:25.999405 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:25.999493 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:25.999658 (+ 165us) server_negotiation.cc:299] Negotiation successful
0621 07:47:25.999729 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260621 07:47:26.000108 3820 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48912
I20260621 07:47:26.000834 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 3882
I20260621 07:47:26.006280 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 4016
I20260621 07:47:26.010813 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 4150
I20260621 07:47:26.015594 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 3788
2026-06-21T07:47:26Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1964 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:47:26 dist-test-slave-lzx3 krb5kdc[4291](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:47:26 dist-test-slave-lzx3 krb5kdc[4291](info): set up 2 sockets
Jun 21 07:47:26 dist-test-slave-lzx3 krb5kdc[4291](info): commencing operation
krb5kdc: starting...
W20260621 07:47:26.968530 771 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/krb5kdc/test-user.keytab.
Jun 21 07:47:27 dist-test-slave-lzx3 krb5kdc[4291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:47:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:47:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.192.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
I20260621 07:47:27.073143 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:41757
--webserver_interface=127.0.192.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:34175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:41757
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:27.171937 4304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:27.172142 4304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:27.172179 4304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:27.175902 4304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:47:27.175966 4304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:27.175987 4304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:27.176018 4304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:47:27.176035 4304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:47:27.180373 4304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:34175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:41757
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:41757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:27.181116 4304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:27.181840 4304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:27.188923 4309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:27.189060 4304 server_base.cc:1034] running on GCE node
W20260621 07:47:27.188937 4312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:27.189193 4310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:27.189818 4304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:27.190685 4304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:27.191856 4304 hybrid_clock.cc:648] HybridClock initialized: now 1782028047191837 us; error 39 us; skew 500 ppm
Jun 21 07:47:27 dist-test-slave-lzx3 krb5kdc[4291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:27.194353 4304 init.cc:377] Logged in from keytab as kudu/127.0.192.254@KRBTEST.COM (short username kudu)
I20260621 07:47:27.195199 4304 webserver.cc:460] Webserver started at http://127.0.192.254:43083/ using document root <none> and password file <none>
I20260621 07:47:27.195686 4304 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:27.195776 4304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:27.196003 4304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:27.197824 4304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "baf5f9bf9a314de7ac60507eb7b8d3f3"
format_stamp: "Formatted at 2026-06-21 07:47:27 on dist-test-slave-lzx3"
server_key: "c3abcdef9fba84cd2b36faead63e7bb9"
server_key_iv: "17fa3da0db5be64f04aabaddc233866a"
server_key_version: "encryptionkey@0"
I20260621 07:47:27.198320 4304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "baf5f9bf9a314de7ac60507eb7b8d3f3"
format_stamp: "Formatted at 2026-06-21 07:47:27 on dist-test-slave-lzx3"
server_key: "c3abcdef9fba84cd2b36faead63e7bb9"
server_key_iv: "17fa3da0db5be64f04aabaddc233866a"
server_key_version: "encryptionkey@0"
I20260621 07:47:27.201263 4304 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:47:27.203140 4319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:27.204049 4304 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:27.204166 4304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "baf5f9bf9a314de7ac60507eb7b8d3f3"
format_stamp: "Formatted at 2026-06-21 07:47:27 on dist-test-slave-lzx3"
server_key: "c3abcdef9fba84cd2b36faead63e7bb9"
server_key_iv: "17fa3da0db5be64f04aabaddc233866a"
server_key_version: "encryptionkey@0"
I20260621 07:47:27.204254 4304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:27.229538 4304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:27.231752 4304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:27.231949 4304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:27.238361 4304 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:41757
I20260621 07:47:27.238397 4371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:41757 every 8 connection(s)
I20260621 07:47:27.239092 4304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:47:27.241247 4372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:27.245685 4372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baf5f9bf9a314de7ac60507eb7b8d3f3: Bootstrap starting.
I20260621 07:47:27.247671 4372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P baf5f9bf9a314de7ac60507eb7b8d3f3: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:27.248067 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 4304
I20260621 07:47:27.248159 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:47:27.248346 771 external_mini_cluster.cc:1428] Setting key e981e7c5b590aee7011cd0c0fc145193
I20260621 07:47:27.248353 4372 log.cc:824] T 00000000000000000000000000000000 P baf5f9bf9a314de7ac60507eb7b8d3f3: Log is configured to *not* fsync() on all Append() calls
I20260621 07:47:27.250080 4372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baf5f9bf9a314de7ac60507eb7b8d3f3: No bootstrap required, opened a new log
I20260621 07:47:27.253664 4372 raft_consensus.cc:348] T 00000000000000000000000000000000 P baf5f9bf9a314de7ac60507eb7b8d3f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baf5f9bf9a314de7ac60507eb7b8d3f3" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 41757 } }
I20260621 07:47:27.253873 4372 raft_consensus.cc:374] T 00000000000000000000000000000000 P baf5f9bf9a314de7ac60507eb7b8d3f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:27.253949 4372 raft_consensus.cc:729] T 00000000000000000000000000000000 P baf5f9bf9a314de7ac60507eb7b8d3f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baf5f9bf9a314de7ac60507eb7b8d3f3, State: Initialized, Role: FOLLOWER
I20260621 07:47:27.254393 4372 consensus_queue.cc:260] T 00000000000000000000000000000000 P baf5f9bf9a314de7ac60507eb7b8d3f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baf5f9bf9a314de7ac60507eb7b8d3f3" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 41757 } }
I20260621 07:47:27.254566 4372 raft_consensus.cc:388] T 00000000000000000000000000000000 P baf5f9bf9a314de7ac60507eb7b8d3f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:47:27.254629 4372 raft_consensus.cc:482] T 00000000000000000000000000000000 P baf5f9bf9a314de7ac60507eb7b8d3f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:47:27.254738 4372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P baf5f9bf9a314de7ac60507eb7b8d3f3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:27.255820 4372 raft_consensus.cc:504] T 00000000000000000000000000000000 P baf5f9bf9a314de7ac60507eb7b8d3f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baf5f9bf9a314de7ac60507eb7b8d3f3" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 41757 } }
Jun 21 07:47:27 dist-test-slave-lzx3 krb5kdc[4291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:47:27.256114 4372 leader_election.cc:302] T 00000000000000000000000000000000 P baf5f9bf9a314de7ac60507eb7b8d3f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: baf5f9bf9a314de7ac60507eb7b8d3f3; no voters:
I20260621 07:47:27.256364 4372 leader_election.cc:288] T 00000000000000000000000000000000 P baf5f9bf9a314de7ac60507eb7b8d3f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:47:27.256417 4377 raft_consensus.cc:2781] T 00000000000000000000000000000000 P baf5f9bf9a314de7ac60507eb7b8d3f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:47:27.256592 4377 raft_consensus.cc:686] T 00000000000000000000000000000000 P baf5f9bf9a314de7ac60507eb7b8d3f3 [term 1 LEADER]: Becoming Leader. State: Replica: baf5f9bf9a314de7ac60507eb7b8d3f3, State: Running, Role: LEADER
I20260621 07:47:27.256848 4377 consensus_queue.cc:237] T 00000000000000000000000000000000 P baf5f9bf9a314de7ac60507eb7b8d3f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baf5f9bf9a314de7ac60507eb7b8d3f3" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 41757 } }
I20260621 07:47:27.257418 4372 sys_catalog.cc:564] T 00000000000000000000000000000000 P baf5f9bf9a314de7ac60507eb7b8d3f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:47:27.257887 4379 sys_catalog.cc:455] T 00000000000000000000000000000000 P baf5f9bf9a314de7ac60507eb7b8d3f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader baf5f9bf9a314de7ac60507eb7b8d3f3. Latest consensus state: current_term: 1 leader_uuid: "baf5f9bf9a314de7ac60507eb7b8d3f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baf5f9bf9a314de7ac60507eb7b8d3f3" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 41757 } } }
I20260621 07:47:27.258517 4379 sys_catalog.cc:458] T 00000000000000000000000000000000 P baf5f9bf9a314de7ac60507eb7b8d3f3 [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:27.258400 4378 sys_catalog.cc:455] T 00000000000000000000000000000000 P baf5f9bf9a314de7ac60507eb7b8d3f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "baf5f9bf9a314de7ac60507eb7b8d3f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baf5f9bf9a314de7ac60507eb7b8d3f3" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 41757 } } }
I20260621 07:47:27.258809 4378 sys_catalog.cc:458] T 00000000000000000000000000000000 P baf5f9bf9a314de7ac60507eb7b8d3f3 [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:27.259614 4386 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:47:27.261392 4376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:27.249312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58940 (local address 127.0.192.254:41757)
0621 07:47:27.249650 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:27.249673 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:27.249704 (+ 31us) server_negotiation.cc:407] Connection header received
0621 07:47:27.250263 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:27.250278 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:27.250526 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:27.250744 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:27.251535 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:27.252302 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:27.253467 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:27.253922 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:27.254124 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:27.256807 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:27.256812 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:27.256826 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:27.256856 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:27.258720 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:27.259194 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:27.259199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:27.259204 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:27.259341 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:27.259636 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:27.259639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:27.259640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:27.260034 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:27.260204 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:27.260383 (+ 179us) server_negotiation.cc:299] Negotiation successful
0621 07:47:27.260644 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":126,"threads_started":1}
I20260621 07:47:27.262678 4386 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:47:27.267306 4386 catalog_manager.cc:1347] Generated new cluster ID: 8115182e099245149c99927dd819ae1f
I20260621 07:47:27.267361 4386 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:47:27.279184 4386 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:47:27.279906 4386 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:47:27.286603 4386 catalog_manager.cc:5868] T 00000000000000000000000000000000 P baf5f9bf9a314de7ac60507eb7b8d3f3: Generated new TSK 0
I20260621 07:47:27.287103 4386 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.192.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab.
Entry for principal kudu/127.0.192.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab.
I20260621 07:47:27.312302 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.193:0
--local_ip_for_outbound_sockets=127.0.192.193
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:41757
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:34175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:27.411495 4398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:27.411720 4398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:27.411782 4398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:27.415618 4398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:27.415702 4398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:27.415789 4398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.193
I20260621 07:47:27.420140 4398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:34175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:41757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.193
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:27.420953 4398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:27.421731 4398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:27.429334 4404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:27.429338 4406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:27.429392 4403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:27.429484 4398 server_base.cc:1034] running on GCE node
I20260621 07:47:27.429811 4398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:27.430294 4398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:27.431435 4398 hybrid_clock.cc:648] HybridClock initialized: now 1782028047431416 us; error 30 us; skew 500 ppm
Jun 21 07:47:27 dist-test-slave-lzx3 krb5kdc[4291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:27.433756 4398 init.cc:377] Logged in from keytab as kudu/127.0.192.193@KRBTEST.COM (short username kudu)
I20260621 07:47:27.434522 4398 webserver.cc:460] Webserver started at http://127.0.192.193:43397/ using document root <none> and password file <none>
I20260621 07:47:27.435009 4398 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:27.435099 4398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:27.435309 4398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:27.437111 4398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-0/data/instance:
uuid: "cf88c1485d36430982ede76e5a574b98"
format_stamp: "Formatted at 2026-06-21 07:47:27 on dist-test-slave-lzx3"
server_key: "d2e659ed40b0fa458addd21208ec70e6"
server_key_iv: "e38ed5a807798ea4729fdbdbf682ead3"
server_key_version: "encryptionkey@0"
I20260621 07:47:27.437593 4398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-0/wal/instance:
uuid: "cf88c1485d36430982ede76e5a574b98"
format_stamp: "Formatted at 2026-06-21 07:47:27 on dist-test-slave-lzx3"
server_key: "d2e659ed40b0fa458addd21208ec70e6"
server_key_iv: "e38ed5a807798ea4729fdbdbf682ead3"
server_key_version: "encryptionkey@0"
I20260621 07:47:27.440696 4398 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:47:27.442605 4413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:27.443542 4398 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:27.443660 4398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-0/wal
uuid: "cf88c1485d36430982ede76e5a574b98"
format_stamp: "Formatted at 2026-06-21 07:47:27 on dist-test-slave-lzx3"
server_key: "d2e659ed40b0fa458addd21208ec70e6"
server_key_iv: "e38ed5a807798ea4729fdbdbf682ead3"
server_key_version: "encryptionkey@0"
I20260621 07:47:27.443763 4398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:27.457628 4398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:27.460016 4398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:27.460206 4398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:27.460696 4398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:47:27.461357 4398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:47:27.461422 4398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:27.461483 4398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:47:27.461513 4398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:27.468918 4398 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.193:38369
I20260621 07:47:27.468935 4526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.193:38369 every 8 connection(s)
I20260621 07:47:27.469600 4398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
I20260621 07:47:27.469863 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 4398
I20260621 07:47:27.469986 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-0/wal/instance
I20260621 07:47:27.470239 771 external_mini_cluster.cc:1428] Setting key f8cc73c76a9ad06fa0f7f83822c65acc
Jun 21 07:47:27 dist-test-slave-lzx3 krb5kdc[4291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.192.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.194@KRBTEST.COM" created.
I20260621 07:47:27.481971 4376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:27.470988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:35097 (local address 127.0.192.254:41757)
0621 07:47:27.471106 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:27.471111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:27.471933 (+ 822us) server_negotiation.cc:407] Connection header received
0621 07:47:27.473588 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:27.473593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:27.473650 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:27.473748 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:27.474703 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:27.475639 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:27.476683 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:27.477053 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:27.477192 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:27.479118 (+ 1926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:27.479124 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:27.479130 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:27.479161 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:27.480307 (+ 1146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:27.480741 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:27.480743 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:27.480744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:27.480838 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:27.481095 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:27.481096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:27.481097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:27.481249 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:27.481361 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:27.481790 (+ 429us) server_negotiation.cc:299] Negotiation successful
0621 07:47:27.481874 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260621 07:47:27.482946 4529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:27.471346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:41757 (local address 127.0.192.193:35097)
0621 07:47:27.471773 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:27.471823 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:27.473396 (+ 1573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:27.473871 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:27.473877 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:27.474175 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:27.474572 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:27.474582 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:27.475757 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:27.475759 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:27.476521 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:27.476529 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:27.477205 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:27.477207 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:27.477324 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:27.477778 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:27.477790 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:27.478945 (+ 1155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:27.480399 (+ 1454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:27.480403 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:27.480411 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:27.480673 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:27.480920 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:27.480922 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:27.480923 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:27.481039 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:27.481365 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:27.481368 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:27.481568 (+ 200us) client_negotiation.cc:769] Sending connection context
0621 07:47:27.481727 (+ 159us) client_negotiation.cc:240] Negotiation successful
0621 07:47:27.482021 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":106,"threads_started":1}
I20260621 07:47:27.483896 4527 heartbeater.cc:347] Connected to a master server at 127.0.192.254:41757
I20260621 07:47:27.484081 4527 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:27.484452 4527 heartbeater.cc:511] Master 127.0.192.254:41757 requested a full tablet report, sending...
I20260621 07:47:27.485751 4336 ts_manager.cc:194] Registered new tserver with Master: cf88c1485d36430982ede76e5a574b98 (127.0.192.193:38369)
I20260621 07:47:27.486836 4336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.193@KRBTEST.COM'} at 127.0.192.193:35097
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab.
Entry for principal kudu/127.0.192.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab.
I20260621 07:47:27.495514 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.194:0
--local_ip_for_outbound_sockets=127.0.192.194
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:41757
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:34175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:27.593566 4532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:27.593786 4532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:27.593824 4532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:27.597646 4532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:27.597687 4532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:27.597736 4532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.194
I20260621 07:47:27.602506 4532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:34175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:41757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.194
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:27.603314 4532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:27.604111 4532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:27.612345 4540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:27.612346 4538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:27.612402 4532 server_base.cc:1034] running on GCE node
W20260621 07:47:27.612392 4537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:27.612885 4532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:27.613454 4532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:27.614639 4532 hybrid_clock.cc:648] HybridClock initialized: now 1782028047614627 us; error 23 us; skew 500 ppm
Jun 21 07:47:27 dist-test-slave-lzx3 krb5kdc[4291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:27.617178 4532 init.cc:377] Logged in from keytab as kudu/127.0.192.194@KRBTEST.COM (short username kudu)
I20260621 07:47:27.617898 4532 webserver.cc:460] Webserver started at http://127.0.192.194:37389/ using document root <none> and password file <none>
I20260621 07:47:27.618319 4532 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:27.618367 4532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:27.618499 4532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:27.620157 4532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-1/data/instance:
uuid: "789e16a812ca48968268687f2c3d9c49"
format_stamp: "Formatted at 2026-06-21 07:47:27 on dist-test-slave-lzx3"
server_key: "52cacb9cbdc92a72a7654c6c1c21b823"
server_key_iv: "722691792bfa2eec3fb10218583a1c8a"
server_key_version: "encryptionkey@0"
I20260621 07:47:27.620626 4532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-1/wal/instance:
uuid: "789e16a812ca48968268687f2c3d9c49"
format_stamp: "Formatted at 2026-06-21 07:47:27 on dist-test-slave-lzx3"
server_key: "52cacb9cbdc92a72a7654c6c1c21b823"
server_key_iv: "722691792bfa2eec3fb10218583a1c8a"
server_key_version: "encryptionkey@0"
I20260621 07:47:27.623659 4532 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260621 07:47:27.625695 4547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:27.626891 4532 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 07:47:27.626969 4532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-1/wal
uuid: "789e16a812ca48968268687f2c3d9c49"
format_stamp: "Formatted at 2026-06-21 07:47:27 on dist-test-slave-lzx3"
server_key: "52cacb9cbdc92a72a7654c6c1c21b823"
server_key_iv: "722691792bfa2eec3fb10218583a1c8a"
server_key_version: "encryptionkey@0"
I20260621 07:47:27.627058 4532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:27.642707 4532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:27.644953 4532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:27.645089 4532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:27.645520 4532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:47:27.646198 4532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:47:27.646242 4532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:27.646274 4532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:47:27.646296 4532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:27.654040 4532 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.194:44003
I20260621 07:47:27.654111 4660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.194:44003 every 8 connection(s)
I20260621 07:47:27.654850 4532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
I20260621 07:47:27.660099 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 4532
I20260621 07:47:27.660248 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-1/wal/instance
I20260621 07:47:27.660537 771 external_mini_cluster.cc:1428] Setting key 78e0e1b697e300588d4f6646360b9209
Jun 21 07:47:27 dist-test-slave-lzx3 krb5kdc[4291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:47:27.667894 4376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:27.656423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:33283 (local address 127.0.192.254:41757)
0621 07:47:27.656598 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:27.656603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:27.657262 (+ 659us) server_negotiation.cc:407] Connection header received
0621 07:47:27.658573 (+ 1311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:27.658575 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:27.658618 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:27.658687 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:27.659690 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:27.660163 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:27.661378 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:27.661654 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:27.661743 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:27.664212 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:27.664217 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:27.664222 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:27.664252 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:27.665919 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:27.666521 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:27.666523 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:27.666524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:27.666584 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:27.666932 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:27.666934 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:27.666935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:27.667079 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:27.667142 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:27.667701 (+ 559us) server_negotiation.cc:299] Negotiation successful
0621 07:47:27.667777 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 07:47:27.668856 4663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:27.656727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:41757 (local address 127.0.192.194:33283)
0621 07:47:27.657101 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:27.657168 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:27.658390 (+ 1222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:27.658830 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:27.658849 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:27.659190 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:27.659579 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:27.659590 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:27.660300 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:27.660305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:27.661271 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:27.661280 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:27.661797 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:27.661801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:27.661952 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:27.662442 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:27.662480 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:27.664064 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:27.666090 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:27.666098 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:27.666111 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:27.666429 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:27.666671 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:27.666675 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:27.666679 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:27.666844 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:27.667195 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:27.667204 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:27.667471 (+ 267us) client_negotiation.cc:769] Sending connection context
0621 07:47:27.667663 (+ 192us) client_negotiation.cc:240] Negotiation successful
0621 07:47:27.667964 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":91,"threads_started":1}
WARNING: no policy specified for kudu/127.0.192.195@KRBTEST.COM; defaulting to no policy
I20260621 07:47:27.669865 4661 heartbeater.cc:347] Connected to a master server at 127.0.192.254:41757
I20260621 07:47:27.670084 4661 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:27.670456 4661 heartbeater.cc:511] Master 127.0.192.254:41757 requested a full tablet report, sending...
I20260621 07:47:27.671434 4336 ts_manager.cc:194] Registered new tserver with Master: 789e16a812ca48968268687f2c3d9c49 (127.0.192.194:44003)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.195@KRBTEST.COM" created.
I20260621 07:47:27.672091 4336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.194@KRBTEST.COM'} at 127.0.192.194:33283
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab.
Entry for principal kudu/127.0.192.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab.
I20260621 07:47:27.689397 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.195:0
--local_ip_for_outbound_sockets=127.0.192.195
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:41757
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:34175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:27.791805 4666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:27.792009 4666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:27.792045 4666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:27.796104 4666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:27.796152 4666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:27.796205 4666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.195
I20260621 07:47:27.801245 4666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:34175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:41757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.195
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:27.802160 4666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:27.803068 4666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:27.812166 4671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:27.812188 4674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:27.812188 4672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:27.812323 4666 server_base.cc:1034] running on GCE node
I20260621 07:47:27.812824 4666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:27.813477 4666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:27.814646 4666 hybrid_clock.cc:648] HybridClock initialized: now 1782028047814628 us; error 28 us; skew 500 ppm
Jun 21 07:47:27 dist-test-slave-lzx3 krb5kdc[4291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:27.817222 4666 init.cc:377] Logged in from keytab as kudu/127.0.192.195@KRBTEST.COM (short username kudu)
I20260621 07:47:27.818058 4666 webserver.cc:460] Webserver started at http://127.0.192.195:38105/ using document root <none> and password file <none>
I20260621 07:47:27.818583 4666 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:27.818672 4666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:27.818897 4666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:27.820829 4666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-2/data/instance:
uuid: "2bf23c4a85354f60a971f387afddf23b"
format_stamp: "Formatted at 2026-06-21 07:47:27 on dist-test-slave-lzx3"
server_key: "98113fc145b1673aa2a487197d8a20ea"
server_key_iv: "bea55f801f84528688817771627bed89"
server_key_version: "encryptionkey@0"
I20260621 07:47:27.821352 4666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-2/wal/instance:
uuid: "2bf23c4a85354f60a971f387afddf23b"
format_stamp: "Formatted at 2026-06-21 07:47:27 on dist-test-slave-lzx3"
server_key: "98113fc145b1673aa2a487197d8a20ea"
server_key_iv: "bea55f801f84528688817771627bed89"
server_key_version: "encryptionkey@0"
I20260621 07:47:27.824862 4666 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:47:27.827081 4681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:27.828140 4666 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:27.828266 4666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-2/wal
uuid: "2bf23c4a85354f60a971f387afddf23b"
format_stamp: "Formatted at 2026-06-21 07:47:27 on dist-test-slave-lzx3"
server_key: "98113fc145b1673aa2a487197d8a20ea"
server_key_iv: "bea55f801f84528688817771627bed89"
server_key_version: "encryptionkey@0"
I20260621 07:47:27.828368 4666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:27.878777 4666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:27.881247 4666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:27.881429 4666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:27.881950 4666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:47:27.882660 4666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:47:27.882727 4666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:27.882787 4666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:47:27.882831 4666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:27.891530 4666 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.195:46487
I20260621 07:47:27.891579 4794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.195:46487 every 8 connection(s)
I20260621 07:47:27.892267 4666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
I20260621 07:47:27.894663 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 4666
I20260621 07:47:27.894766 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782028028522580-771-0/minicluster-data/ts-2/wal/instance
I20260621 07:47:27.895090 771 external_mini_cluster.cc:1428] Setting key b23b15eb6f9b4d10888ead3357a00ac0
Jun 21 07:47:27 dist-test-slave-lzx3 krb5kdc[4291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:47:27.903894 4376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:27.893830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:57999 (local address 127.0.192.254:41757)
0621 07:47:27.893963 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:27.893967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:27.894691 (+ 724us) server_negotiation.cc:407] Connection header received
0621 07:47:27.896264 (+ 1573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:27.896266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:27.896315 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:27.896390 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:27.897412 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:27.897871 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:27.898880 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:27.899074 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:27.899175 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:27.901154 (+ 1979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:27.901157 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:27.901159 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:27.901185 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:27.902186 (+ 1001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:27.902630 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:27.902635 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:27.902636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:27.902707 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:27.903007 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:27.903011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:27.903012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:27.903161 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:27.903259 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:27.903664 (+ 405us) server_negotiation.cc:299] Negotiation successful
0621 07:47:27.903756 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 07:47:27.904498 4797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:27.894108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:41757 (local address 127.0.192.195:57999)
0621 07:47:27.894517 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:27.894555 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:27.896060 (+ 1505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:27.896566 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:27.896586 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:27.896892 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:27.897279 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:27.897289 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:27.897986 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:27.897990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:27.898761 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:27.898769 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:27.899203 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:27.899206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:27.899349 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:27.899733 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:27.899746 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:27.900999 (+ 1253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:27.902291 (+ 1292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:27.902296 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:27.902307 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:27.902525 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:27.902795 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:27.902798 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:27.902800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:27.902918 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:27.903256 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:27.903261 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:27.903456 (+ 195us) client_negotiation.cc:769] Sending connection context
0621 07:47:27.903615 (+ 159us) client_negotiation.cc:240] Negotiation successful
0621 07:47:27.903864 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":102,"threads_started":1}
I20260621 07:47:27.905519 4795 heartbeater.cc:347] Connected to a master server at 127.0.192.254:41757
I20260621 07:47:27.905746 4795 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:27.906137 4795 heartbeater.cc:511] Master 127.0.192.254:41757 requested a full tablet report, sending...
I20260621 07:47:27.906922 4336 ts_manager.cc:194] Registered new tserver with Master: 2bf23c4a85354f60a971f387afddf23b (127.0.192.195:46487)
I20260621 07:47:27.907506 4336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.195@KRBTEST.COM'} at 127.0.192.195:57999
I20260621 07:47:27.908640 771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 21 07:47:27 dist-test-slave-lzx3 krb5kdc[4291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 07:47:27 dist-test-slave-lzx3 krb5kdc[4291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.192.193@KRBTEST.COM
I20260621 07:47:27.930317 4802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:27.922408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53446 (local address 127.0.192.193:38369)
0621 07:47:27.922680 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:27.922688 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:27.922706 (+ 18us) server_negotiation.cc:407] Connection header received
0621 07:47:27.922743 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:27.922747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:27.922870 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:27.922961 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:27.923489 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:27.924117 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:27.924894 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:27.925219 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:27.925302 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:27.927321 (+ 2019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:27.927328 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:27.927336 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:27.927356 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:27.928786 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:27.929172 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:27.929180 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:27.929181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:27.929247 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:27.929493 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:27.929497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:27.929498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:27.929690 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:27.929867 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:27.930040 (+ 173us) server_negotiation.cc:299] Negotiation successful
0621 07:47:27.930166 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":89,"threads_started":1}
Jun 21 07:47:27 dist-test-slave-lzx3 krb5kdc[4291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:47:27.939456 4376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:27.932518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58954 (local address 127.0.192.254:41757)
0621 07:47:27.932655 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:27.932662 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:27.932719 (+ 57us) server_negotiation.cc:407] Connection header received
0621 07:47:27.932819 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:27.932820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:27.932849 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:27.932903 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:27.933389 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:27.933827 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:27.934491 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:27.934684 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:27.934750 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:27.936895 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:27.936897 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:27.936899 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:27.936917 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:27.937976 (+ 1059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:27.938377 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:27.938380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:27.938381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:27.938442 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:27.938727 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:27.938729 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:27.938730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:27.938895 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:27.938987 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:27.939205 (+ 218us) server_negotiation.cc:299] Negotiation successful
0621 07:47:27.939278 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260621 07:47:27.939834 4336 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58954
I20260621 07:47:27.940356 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 4398
I20260621 07:47:27.945459 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 4532
I20260621 07:47:27.950150 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 4666
I20260621 07:47:27.954797 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 4304
2026-06-21T07:47:27Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1938 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:47:28 dist-test-slave-lzx3 krb5kdc[4807](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:47:28 dist-test-slave-lzx3 krb5kdc[4807](info): set up 2 sockets
Jun 21 07:47:28 dist-test-slave-lzx3 krb5kdc[4807](info): commencing operation
krb5kdc: starting...
W20260621 07:47:28.904274 771 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/krb5kdc/test-user.keytab.
Jun 21 07:47:28 dist-test-slave-lzx3 krb5kdc[4807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:47:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:47:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.192.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
I20260621 07:47:29.019260 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:33209
--webserver_interface=127.0.192.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:39785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:33209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:29.121712 4820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:29.121918 4820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:29.121991 4820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:29.125854 4820 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:47:29.125901 4820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:29.125921 4820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:29.125979 4820 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:47:29.126015 4820 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:47:29.130625 4820 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:39785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:33209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:33209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:29.131511 4820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:29.132418 4820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:29.139993 4828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:29.139959 4825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:29.140014 4826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:29.140326 4820 server_base.cc:1034] running on GCE node
I20260621 07:47:29.140856 4820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:29.141782 4820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:29.142930 4820 hybrid_clock.cc:648] HybridClock initialized: now 1782028049142905 us; error 36 us; skew 500 ppm
Jun 21 07:47:29 dist-test-slave-lzx3 krb5kdc[4807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028049, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:29.145550 4820 init.cc:377] Logged in from keytab as kudu/127.0.192.254@KRBTEST.COM (short username kudu)
I20260621 07:47:29.146245 4820 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:29.146360 4820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:29.146561 4820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:29.148372 4820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "f44e4c4fc10b4cf183b6b64b5fd42571"
format_stamp: "Formatted at 2026-06-21 07:47:29 on dist-test-slave-lzx3"
server_key: "fb25f5dd43fecb2a4223b05432e575eb"
server_key_iv: "120e10ea975b78bcd605ef069bd3c023"
server_key_version: "encryptionkey@0"
I20260621 07:47:29.148919 4820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "f44e4c4fc10b4cf183b6b64b5fd42571"
format_stamp: "Formatted at 2026-06-21 07:47:29 on dist-test-slave-lzx3"
server_key: "fb25f5dd43fecb2a4223b05432e575eb"
server_key_iv: "120e10ea975b78bcd605ef069bd3c023"
server_key_version: "encryptionkey@0"
I20260621 07:47:29.151999 4820 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:47:29.154294 4834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:29.155377 4820 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260621 07:47:29.155514 4820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "f44e4c4fc10b4cf183b6b64b5fd42571"
format_stamp: "Formatted at 2026-06-21 07:47:29 on dist-test-slave-lzx3"
server_key: "fb25f5dd43fecb2a4223b05432e575eb"
server_key_iv: "120e10ea975b78bcd605ef069bd3c023"
server_key_version: "encryptionkey@0"
I20260621 07:47:29.155616 4820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:29.173862 4820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:29.176932 4820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:29.177114 4820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:29.183079 4820 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:33209
I20260621 07:47:29.183314 4886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:33209 every 8 connection(s)
I20260621 07:47:29.183653 4820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:47:29.184376 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 4820
I20260621 07:47:29.184507 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:47:29.184820 771 external_mini_cluster.cc:1428] Setting key d10fdff769d4e10068099a7e18cf5fc1
I20260621 07:47:29.186554 4887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:29.192452 4887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f44e4c4fc10b4cf183b6b64b5fd42571: Bootstrap starting.
Jun 21 07:47:29 dist-test-slave-lzx3 krb5kdc[4807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:47:29.195084 4887 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f44e4c4fc10b4cf183b6b64b5fd42571: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:29.195947 4887 log.cc:824] T 00000000000000000000000000000000 P f44e4c4fc10b4cf183b6b64b5fd42571: Log is configured to *not* fsync() on all Append() calls
I20260621 07:47:29.197860 4887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f44e4c4fc10b4cf183b6b64b5fd42571: No bootstrap required, opened a new log
I20260621 07:47:29.198971 4890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:29.186009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50048 (local address 127.0.192.254:33209)
0621 07:47:29.186403 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:29.186428 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:29.186458 (+ 30us) server_negotiation.cc:407] Connection header received
0621 07:47:29.187077 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:29.187092 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:29.187388 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:29.187637 (+ 249us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:29.188414 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:29.189349 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:29.190509 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:29.191044 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:29.191162 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:29.194053 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:29.194059 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:29.194073 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:29.194100 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:29.196121 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:29.196692 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:29.196697 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:29.196703 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:29.196813 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:29.197100 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:29.197102 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:29.197104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:29.197518 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:29.197703 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:29.197871 (+ 168us) server_negotiation.cc:299] Negotiation successful
0621 07:47:29.198121 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":152,"threads_started":1}
I20260621 07:47:29.201140 4887 raft_consensus.cc:348] T 00000000000000000000000000000000 P f44e4c4fc10b4cf183b6b64b5fd42571 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f44e4c4fc10b4cf183b6b64b5fd42571" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 33209 } }
I20260621 07:47:29.201319 4887 raft_consensus.cc:374] T 00000000000000000000000000000000 P f44e4c4fc10b4cf183b6b64b5fd42571 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:29.201356 4887 raft_consensus.cc:729] T 00000000000000000000000000000000 P f44e4c4fc10b4cf183b6b64b5fd42571 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f44e4c4fc10b4cf183b6b64b5fd42571, State: Initialized, Role: FOLLOWER
I20260621 07:47:29.201643 4887 consensus_queue.cc:260] T 00000000000000000000000000000000 P f44e4c4fc10b4cf183b6b64b5fd42571 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f44e4c4fc10b4cf183b6b64b5fd42571" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 33209 } }
I20260621 07:47:29.201732 4887 raft_consensus.cc:388] T 00000000000000000000000000000000 P f44e4c4fc10b4cf183b6b64b5fd42571 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:47:29.201763 4887 raft_consensus.cc:482] T 00000000000000000000000000000000 P f44e4c4fc10b4cf183b6b64b5fd42571 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:47:29.201819 4887 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f44e4c4fc10b4cf183b6b64b5fd42571 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:29.231274 4887 raft_consensus.cc:504] T 00000000000000000000000000000000 P f44e4c4fc10b4cf183b6b64b5fd42571 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f44e4c4fc10b4cf183b6b64b5fd42571" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 33209 } }
I20260621 07:47:29.231688 4887 leader_election.cc:302] T 00000000000000000000000000000000 P f44e4c4fc10b4cf183b6b64b5fd42571 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f44e4c4fc10b4cf183b6b64b5fd42571; no voters:
I20260621 07:47:29.232038 4887 leader_election.cc:288] T 00000000000000000000000000000000 P f44e4c4fc10b4cf183b6b64b5fd42571 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:47:29.232204 4892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f44e4c4fc10b4cf183b6b64b5fd42571 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:47:29.232419 4892 raft_consensus.cc:686] T 00000000000000000000000000000000 P f44e4c4fc10b4cf183b6b64b5fd42571 [term 1 LEADER]: Becoming Leader. State: Replica: f44e4c4fc10b4cf183b6b64b5fd42571, State: Running, Role: LEADER
I20260621 07:47:29.232795 4892 consensus_queue.cc:237] T 00000000000000000000000000000000 P f44e4c4fc10b4cf183b6b64b5fd42571 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f44e4c4fc10b4cf183b6b64b5fd42571" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 33209 } }
I20260621 07:47:29.234207 4893 sys_catalog.cc:455] T 00000000000000000000000000000000 P f44e4c4fc10b4cf183b6b64b5fd42571 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f44e4c4fc10b4cf183b6b64b5fd42571" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f44e4c4fc10b4cf183b6b64b5fd42571" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 33209 } } }
I20260621 07:47:29.234360 4893 sys_catalog.cc:458] T 00000000000000000000000000000000 P f44e4c4fc10b4cf183b6b64b5fd42571 [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:29.234347 4894 sys_catalog.cc:455] T 00000000000000000000000000000000 P f44e4c4fc10b4cf183b6b64b5fd42571 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f44e4c4fc10b4cf183b6b64b5fd42571. Latest consensus state: current_term: 1 leader_uuid: "f44e4c4fc10b4cf183b6b64b5fd42571" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f44e4c4fc10b4cf183b6b64b5fd42571" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 33209 } } }
I20260621 07:47:29.234475 4894 sys_catalog.cc:458] T 00000000000000000000000000000000 P f44e4c4fc10b4cf183b6b64b5fd42571 [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:29.234602 4887 sys_catalog.cc:564] T 00000000000000000000000000000000 P f44e4c4fc10b4cf183b6b64b5fd42571 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:47:29.237208 4896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:47:29.241148 4896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:47:29.246649 4896 catalog_manager.cc:1347] Generated new cluster ID: 6174d01908804562ba48b7080b371bba
I20260621 07:47:29.246754 4896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:47:29.269905 4896 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:47:29.270994 4896 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:47:29.282563 4896 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f44e4c4fc10b4cf183b6b64b5fd42571: Generated new TSK 0
I20260621 07:47:29.283248 4896 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.192.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab.
Entry for principal kudu/127.0.192.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab.
W20260621 07:47:30.288497 771 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.192.193: real 0.994s user 0.001s sys 0.002s
I20260621 07:47:30.289045 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.193:0
--local_ip_for_outbound_sockets=127.0.192.193
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:33209
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:39785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:30.395455 4913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:30.395735 4913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:30.395829 4913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:30.399595 4913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:30.399642 4913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:30.399748 4913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.193
I20260621 07:47:30.404484 4913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:39785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:33209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.193
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:30.405395 4913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:30.406306 4913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:30.414687 4919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:30.414683 4918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:30.414683 4921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:30.415086 4913 server_base.cc:1034] running on GCE node
I20260621 07:47:30.415423 4913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:30.415894 4913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:30.417042 4913 hybrid_clock.cc:648] HybridClock initialized: now 1782028050417029 us; error 23 us; skew 500 ppm
Jun 21 07:47:30 dist-test-slave-lzx3 krb5kdc[4807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028050, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:30.419631 4913 init.cc:377] Logged in from keytab as kudu/127.0.192.193@KRBTEST.COM (short username kudu)
I20260621 07:47:30.420204 4913 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:30.421006 4913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:30.421187 4913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:30.422859 4913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-0/data/instance:
uuid: "c0c5bd10acb2484981118a3e42e56a11"
format_stamp: "Formatted at 2026-06-21 07:47:30 on dist-test-slave-lzx3"
server_key: "54fcfccf511fe9424b9cbdf4783410b0"
server_key_iv: "f26a5d7080bb89633b102302eabb513c"
server_key_version: "encryptionkey@0"
I20260621 07:47:30.423326 4913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-0/wal/instance:
uuid: "c0c5bd10acb2484981118a3e42e56a11"
format_stamp: "Formatted at 2026-06-21 07:47:30 on dist-test-slave-lzx3"
server_key: "54fcfccf511fe9424b9cbdf4783410b0"
server_key_iv: "f26a5d7080bb89633b102302eabb513c"
server_key_version: "encryptionkey@0"
I20260621 07:47:30.426585 4913 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260621 07:47:30.428649 4927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:30.429724 4913 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260621 07:47:30.429862 4913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-0/wal
uuid: "c0c5bd10acb2484981118a3e42e56a11"
format_stamp: "Formatted at 2026-06-21 07:47:30 on dist-test-slave-lzx3"
server_key: "54fcfccf511fe9424b9cbdf4783410b0"
server_key_iv: "f26a5d7080bb89633b102302eabb513c"
server_key_version: "encryptionkey@0"
I20260621 07:47:30.429968 4913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:30.443612 4913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:30.446610 4913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:30.446807 4913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:30.447341 4913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:47:30.448027 4913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:47:30.448069 4913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:30.448133 4913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:47:30.448163 4913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:30.457437 4913 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.193:33089
I20260621 07:47:30.457450 5040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.193:33089 every 8 connection(s)
I20260621 07:47:30.457953 4913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
I20260621 07:47:30.464370 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 4913
I20260621 07:47:30.464488 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-0/wal/instance
I20260621 07:47:30.464751 771 external_mini_cluster.cc:1428] Setting key 7ed6d6e57b35c36861b697de521e3a9a
Jun 21 07:47:30 dist-test-slave-lzx3 krb5kdc[4807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028050, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:47:30.472084 5043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:30.459603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:33353 (local address 127.0.192.254:33209)
0621 07:47:30.459899 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:30.459903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:30.460579 (+ 676us) server_negotiation.cc:407] Connection header received
0621 07:47:30.461883 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:30.461889 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:30.461965 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:30.462062 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:30.463050 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:30.463932 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:30.465194 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:30.465509 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:30.465613 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:30.468165 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:30.468171 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:30.468174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:30.468205 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:30.470036 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:30.470472 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:30.470478 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:30.470482 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:30.470592 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:30.470896 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:30.470900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:30.470901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:30.471182 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:30.471314 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:30.471694 (+ 380us) server_negotiation.cc:299] Negotiation successful
0621 07:47:30.471832 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":109,"threads_started":1}
I20260621 07:47:30.472500 5044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:30.459984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:33209 (local address 127.0.192.193:33353)
0621 07:47:30.460419 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:30.460459 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:30.461688 (+ 1229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:30.462195 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:30.462201 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:30.462498 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:30.462886 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:30.462898 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:30.464055 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:30.464057 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:30.465070 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:30.465080 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:30.465684 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:30.465687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:30.465846 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:30.466450 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:30.466472 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:30.468006 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:30.470161 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:30.470165 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:30.470173 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:30.470394 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:30.470687 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:30.470689 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:30.470690 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:30.470791 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:30.471292 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:30.471296 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:30.471477 (+ 181us) client_negotiation.cc:769] Sending connection context
0621 07:47:30.471605 (+ 128us) client_negotiation.cc:240] Negotiation successful
0621 07:47:30.471796 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":117,"threads_started":1}
I20260621 07:47:30.473804 5041 heartbeater.cc:347] Connected to a master server at 127.0.192.254:33209
I20260621 07:47:30.474032 5041 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:30.474483 5041 heartbeater.cc:511] Master 127.0.192.254:33209 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.192.194@KRBTEST.COM; defaulting to no policy
I20260621 07:47:30.475939 4851 ts_manager.cc:194] Registered new tserver with Master: c0c5bd10acb2484981118a3e42e56a11 (127.0.192.193:33089)
I20260621 07:47:30.477057 4851 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.193@KRBTEST.COM'} at 127.0.192.193:33353
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.194@KRBTEST.COM" created.
I20260621 07:47:31.478965 5041 heartbeater.cc:503] Master 127.0.192.254:33209 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab.
Entry for principal kudu/127.0.192.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab.
W20260621 07:47:32.055758 771 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.192.194: real 1.589s user 0.004s sys 0.000s
I20260621 07:47:32.056245 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.194:0
--local_ip_for_outbound_sockets=127.0.192.194
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:33209
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:39785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:32.163405 5047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:32.163640 5047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:32.163719 5047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:32.168107 5047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:32.168188 5047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:32.168282 5047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.194
I20260621 07:47:32.173324 5047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:39785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:33209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.194
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:32.174238 5047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:32.175125 5047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:32.183840 5052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:32.183943 5047 server_base.cc:1034] running on GCE node
W20260621 07:47:32.184101 5053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:32.183867 5055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:32.184633 5047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:32.185225 5047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:32.186398 5047 hybrid_clock.cc:648] HybridClock initialized: now 1782028052186385 us; error 27 us; skew 500 ppm
Jun 21 07:47:32 dist-test-slave-lzx3 krb5kdc[4807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:32.192890 5047 init.cc:377] Logged in from keytab as kudu/127.0.192.194@KRBTEST.COM (short username kudu)
I20260621 07:47:32.193575 5047 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:32.193663 5047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:32.193873 5047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:32.195664 5047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-1/data/instance:
uuid: "fa05bdb779cb4da698075d717e9c2d39"
format_stamp: "Formatted at 2026-06-21 07:47:32 on dist-test-slave-lzx3"
server_key: "9e6501d0651ec9f3015707eaa232add3"
server_key_iv: "6138d7ad9dd4bea676f0c1a7044a9e67"
server_key_version: "encryptionkey@0"
I20260621 07:47:32.196197 5047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-1/wal/instance:
uuid: "fa05bdb779cb4da698075d717e9c2d39"
format_stamp: "Formatted at 2026-06-21 07:47:32 on dist-test-slave-lzx3"
server_key: "9e6501d0651ec9f3015707eaa232add3"
server_key_iv: "6138d7ad9dd4bea676f0c1a7044a9e67"
server_key_version: "encryptionkey@0"
I20260621 07:47:32.199688 5047 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 07:47:32.201959 5061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:32.203089 5047 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 07:47:32.203209 5047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-1/wal
uuid: "fa05bdb779cb4da698075d717e9c2d39"
format_stamp: "Formatted at 2026-06-21 07:47:32 on dist-test-slave-lzx3"
server_key: "9e6501d0651ec9f3015707eaa232add3"
server_key_iv: "6138d7ad9dd4bea676f0c1a7044a9e67"
server_key_version: "encryptionkey@0"
I20260621 07:47:32.203306 5047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:32.226037 5047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:32.228315 5047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:32.228474 5047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:32.228940 5047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:47:32.229624 5047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:47:32.229693 5047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:32.229769 5047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:47:32.229810 5047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:32.238837 5047 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.194:39067
I20260621 07:47:32.238862 5174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.194:39067 every 8 connection(s)
I20260621 07:47:32.239318 5047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
I20260621 07:47:32.242646 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 5047
I20260621 07:47:32.242779 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-1/wal/instance
I20260621 07:47:32.243067 771 external_mini_cluster.cc:1428] Setting key b44f2bfa4f34e3d92b7d2dc0881887f9
Jun 21 07:47:32 dist-test-slave-lzx3 krb5kdc[4807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.192.195@KRBTEST.COM; defaulting to no policy
I20260621 07:47:32.252717 5177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:32.240862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:32849 (local address 127.0.192.254:33209)
0621 07:47:32.241102 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:32.241106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:32.241693 (+ 587us) server_negotiation.cc:407] Connection header received
0621 07:47:32.243059 (+ 1366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:32.243062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:32.243123 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:32.243225 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:32.244589 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:32.245104 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:32.246307 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:32.246560 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:32.246679 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:32.249071 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:32.249074 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:32.249076 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:32.249098 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:32.250562 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:32.251061 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:32.251067 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:32.251069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:32.251161 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:32.251558 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:32.251563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:32.251565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:32.251763 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:32.251871 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:32.252359 (+ 488us) server_negotiation.cc:299] Negotiation successful
0621 07:47:32.252489 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":101,"threads_started":1}
I20260621 07:47:32.253455 5178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:32.241140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:33209 (local address 127.0.192.194:32849)
0621 07:47:32.241561 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:32.241599 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:32.242831 (+ 1232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:32.243390 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:32.243396 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:32.243845 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:32.244431 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:32.244445 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:32.245236 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:32.245239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:32.246179 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:32.246189 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:32.246708 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:32.246712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:32.246898 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:32.247447 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:32.247463 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:32.248897 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:32.250680 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:32.250687 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:32.250699 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:32.250953 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:32.251295 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:32.251302 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:32.251305 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:32.251448 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:32.251882 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:32.251891 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:32.252116 (+ 225us) client_negotiation.cc:769] Sending connection context
0621 07:47:32.252298 (+ 182us) client_negotiation.cc:240] Negotiation successful
0621 07:47:32.252641 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":93,"threads_started":1}
I20260621 07:47:32.254546 5175 heartbeater.cc:347] Connected to a master server at 127.0.192.254:33209
I20260621 07:47:32.254755 5175 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:32.255153 5175 heartbeater.cc:511] Master 127.0.192.254:33209 requested a full tablet report, sending...
I20260621 07:47:32.256108 4851 ts_manager.cc:194] Registered new tserver with Master: fa05bdb779cb4da698075d717e9c2d39 (127.0.192.194:39067)
I20260621 07:47:32.256691 4851 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.194@KRBTEST.COM'} at 127.0.192.194:32849
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.195@KRBTEST.COM" created.
I20260621 07:47:33.258682 5175 heartbeater.cc:503] Master 127.0.192.254:33209 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab.
Entry for principal kudu/127.0.192.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab.
W20260621 07:47:33.277232 771 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.192.195: real 1.032s user 0.002s sys 0.001s
I20260621 07:47:33.277746 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.195:0
--local_ip_for_outbound_sockets=127.0.192.195
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:33209
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:39785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:33.385291 5181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:33.385541 5181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:33.385586 5181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:33.389411 5181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:33.389459 5181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:33.389515 5181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.195
I20260621 07:47:33.394388 5181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:39785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:33209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.195
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:33.395231 5181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:33.396164 5181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:33.404970 5189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:33.405009 5186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:33.405016 5187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:33.405314 5181 server_base.cc:1034] running on GCE node
I20260621 07:47:33.405742 5181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:33.406338 5181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:33.407521 5181 hybrid_clock.cc:648] HybridClock initialized: now 1782028053407508 us; error 28 us; skew 500 ppm
Jun 21 07:47:33 dist-test-slave-lzx3 krb5kdc[4807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:33.410147 5181 init.cc:377] Logged in from keytab as kudu/127.0.192.195@KRBTEST.COM (short username kudu)
I20260621 07:47:33.410858 5181 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:33.410948 5181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:33.411147 5181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:33.413151 5181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-2/data/instance:
uuid: "22bd06a8cbfb43519f365be45ade3a94"
format_stamp: "Formatted at 2026-06-21 07:47:33 on dist-test-slave-lzx3"
server_key: "7aa3d2b05d7876beb6b3406f4b618959"
server_key_iv: "c8f6e2a84f690e0d672677e169cddc98"
server_key_version: "encryptionkey@0"
I20260621 07:47:33.413708 5181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-2/wal/instance:
uuid: "22bd06a8cbfb43519f365be45ade3a94"
format_stamp: "Formatted at 2026-06-21 07:47:33 on dist-test-slave-lzx3"
server_key: "7aa3d2b05d7876beb6b3406f4b618959"
server_key_iv: "c8f6e2a84f690e0d672677e169cddc98"
server_key_version: "encryptionkey@0"
I20260621 07:47:33.417186 5181 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:47:33.419399 5195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:33.420457 5181 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260621 07:47:33.420603 5181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-2/wal
uuid: "22bd06a8cbfb43519f365be45ade3a94"
format_stamp: "Formatted at 2026-06-21 07:47:33 on dist-test-slave-lzx3"
server_key: "7aa3d2b05d7876beb6b3406f4b618959"
server_key_iv: "c8f6e2a84f690e0d672677e169cddc98"
server_key_version: "encryptionkey@0"
I20260621 07:47:33.420717 5181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:33.439330 5181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:33.442135 5181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:33.442334 5181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:33.442879 5181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:47:33.443665 5181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:47:33.443738 5181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:33.443806 5181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:47:33.443854 5181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:33.452641 5181 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.195:44939
I20260621 07:47:33.452666 5308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.195:44939 every 8 connection(s)
I20260621 07:47:33.453171 5181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
Jun 21 07:47:33 dist-test-slave-lzx3 krb5kdc[4807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:47:33.463469 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 5181
I20260621 07:47:33.463593 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-2/wal/instance
I20260621 07:47:33.463912 771 external_mini_cluster.cc:1428] Setting key 5089f89a77525c949c996a45614ba373
I20260621 07:47:33.466199 5311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:33.454741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:50065 (local address 127.0.192.254:33209)
0621 07:47:33.455029 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:33.455033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:33.455648 (+ 615us) server_negotiation.cc:407] Connection header received
0621 07:47:33.456953 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:33.456956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:33.457002 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:33.457083 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:33.458255 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:33.458823 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:33.459972 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:33.460216 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:33.460353 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:33.462557 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:33.462561 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:33.462564 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:33.462591 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:33.463982 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:33.464547 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:33.464549 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:33.464550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:33.464707 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:33.465095 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:33.465097 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:33.465098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:33.465326 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:33.465399 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:33.465971 (+ 572us) server_negotiation.cc:299] Negotiation successful
0621 07:47:33.466079 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":94,"threads_started":1}
I20260621 07:47:33.467003 5312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:33.455061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:33209 (local address 127.0.192.195:50065)
0621 07:47:33.455494 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:33.455533 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:33.456763 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:33.457239 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:33.457245 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:33.457564 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:33.458052 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:33.458065 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:33.458980 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:33.458985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:33.459841 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:33.459849 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:33.460382 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:33.460386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:33.460575 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:33.461045 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:33.461060 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:33.462408 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:33.464082 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:33.464086 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:33.464096 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:33.464451 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:33.464809 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:33.464813 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:33.464815 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:33.464980 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:33.465423 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:33.465430 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:33.465722 (+ 292us) client_negotiation.cc:769] Sending connection context
0621 07:47:33.465932 (+ 210us) client_negotiation.cc:240] Negotiation successful
0621 07:47:33.466228 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":103,"threads_started":1}
I20260621 07:47:33.467989 5309 heartbeater.cc:347] Connected to a master server at 127.0.192.254:33209
I20260621 07:47:33.468209 5309 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:33.468669 5309 heartbeater.cc:511] Master 127.0.192.254:33209 requested a full tablet report, sending...
I20260621 07:47:33.469548 4851 ts_manager.cc:194] Registered new tserver with Master: 22bd06a8cbfb43519f365be45ade3a94 (127.0.192.195:44939)
I20260621 07:47:33.470103 4851 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.195@KRBTEST.COM'} at 127.0.192.195:50065
I20260621 07:47:33.477550 771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:47:33.485756 5311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:33.479254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35076 (local address 127.0.192.254:33209)
0621 07:47:33.479355 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:33.479358 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:33.479549 (+ 191us) server_negotiation.cc:407] Connection header received
0621 07:47:33.479631 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:33.479633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:33.479664 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:33.479727 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:33.480278 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:33.480861 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:33.481621 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:33.481833 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:33.481969 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:33.483256 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:33.483258 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:33.483260 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:33.483278 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:33.484501 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:33.484880 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:33.484882 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:33.484883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:33.484944 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:33.485135 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:33.485137 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:33.485138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:33.485309 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:33.485398 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:33.485549 (+ 151us) server_negotiation.cc:299] Negotiation successful
0621 07:47:33.485628 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260621 07:47:33.488350 4851 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35076:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:47:33.491127 4851 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:47:33.504941 5326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:33.500073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:33089 (local address 127.0.0.1:60046)
0621 07:47:33.500292 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:33.500304 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:33.500377 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:33.501272 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:33.501275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:33.501291 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:33.501377 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:33.501383 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:33.502531 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:33.502535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:33.503659 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:33.503667 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:33.504397 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:33.504401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:33.504498 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:33.504529 (+ 31us) client_negotiation.cc:769] Sending connection context
0621 07:47:33.504651 (+ 122us) client_negotiation.cc:240] Negotiation successful
0621 07:47:33.504776 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":14,"thread_start_us":113,"threads_started":1}
I20260621 07:47:33.505091 5322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:33.499076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:39067 (local address 127.0.0.1:47614)
0621 07:47:33.499439 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:33.499489 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:33.499609 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:33.500333 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:33.500340 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:33.500362 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:33.500476 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:33.500482 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:33.502210 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:33.502214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:33.503428 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:33.503436 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:33.504534 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:33.504537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:33.504738 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:33.504795 (+ 57us) client_negotiation.cc:769] Sending connection context
0621 07:47:33.504857 (+ 62us) client_negotiation.cc:240] Negotiation successful
0621 07:47:33.504965 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":132,"thread_start_us":104,"threads_started":1}
I20260621 07:47:33.505162 5325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:33.499888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44248 (local address 127.0.192.195:44939)
0621 07:47:33.500333 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:33.500343 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:33.500361 (+ 18us) server_negotiation.cc:407] Connection header received
0621 07:47:33.500421 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:33.500426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:33.500651 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:33.501026 (+ 375us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:33.501153 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:33.502210 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:33.503701 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:33.504397 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:33.504615 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:33.504756 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:33.504928 (+ 172us) server_negotiation.cc:299] Negotiation successful
0621 07:47:33.505024 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":162,"threads_started":1}
I20260621 07:47:33.505324 5323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:33.499224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47614 (local address 127.0.192.194:39067)
0621 07:47:33.499601 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:33.499608 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:33.499631 (+ 23us) server_negotiation.cc:407] Connection header received
0621 07:47:33.499916 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:33.499925 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:33.500167 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:33.500655 (+ 488us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:33.500811 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:33.502053 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:33.503586 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:33.504397 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:33.504940 (+ 543us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:33.505071 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:33.505114 (+ 43us) server_negotiation.cc:299] Negotiation successful
0621 07:47:33.505206 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":82,"threads_started":1}
I20260621 07:47:33.505539 5324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:33.499733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.195:44939 (local address 127.0.0.1:44248)
0621 07:47:33.499951 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:33.499966 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:33.500048 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:33.500798 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:33.500801 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:33.500817 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:33.500895 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:33.500902 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:33.502335 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:33.502339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:33.503568 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:33.503578 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:33.504534 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:33.504537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:33.504797 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:33.504817 (+ 20us) client_negotiation.cc:769] Sending connection context
0621 07:47:33.504889 (+ 72us) client_negotiation.cc:240] Negotiation successful
0621 07:47:33.505296 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":126,"thread_start_us":105,"threads_started":1}
I20260621 07:47:33.505987 5327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:33.500537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60046 (local address 127.0.192.193:33089)
0621 07:47:33.500937 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:33.500942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:33.500957 (+ 15us) server_negotiation.cc:407] Connection header received
0621 07:47:33.501008 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:33.501011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:33.501126 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:33.501284 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:33.501486 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:33.502403 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:33.503752 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:33.504281 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:33.504396 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:33.504534 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:33.505718 (+ 1184us) server_negotiation.cc:299] Negotiation successful
0621 07:47:33.505819 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":93,"threads_started":1}
I20260621 07:47:33.508142 5109 tablet_service.cc:1467] Processing CreateTablet for tablet 2340240d7b034321b0cb4c0b35396d09 (DEFAULT_TABLE table=test-table [id=ad3fbac0d9294532a2f03e66cafcb178]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:47:33.508149 5243 tablet_service.cc:1467] Processing CreateTablet for tablet 2340240d7b034321b0cb4c0b35396d09 (DEFAULT_TABLE table=test-table [id=ad3fbac0d9294532a2f03e66cafcb178]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:47:33.509150 5109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2340240d7b034321b0cb4c0b35396d09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:33.509207 5243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2340240d7b034321b0cb4c0b35396d09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:33.514815 5328 tablet_bootstrap.cc:492] T 2340240d7b034321b0cb4c0b35396d09 P 22bd06a8cbfb43519f365be45ade3a94: Bootstrap starting.
I20260621 07:47:33.514861 4975 tablet_service.cc:1467] Processing CreateTablet for tablet 2340240d7b034321b0cb4c0b35396d09 (DEFAULT_TABLE table=test-table [id=ad3fbac0d9294532a2f03e66cafcb178]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:47:33.515825 4975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2340240d7b034321b0cb4c0b35396d09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:33.516142 5329 tablet_bootstrap.cc:492] T 2340240d7b034321b0cb4c0b35396d09 P fa05bdb779cb4da698075d717e9c2d39: Bootstrap starting.
I20260621 07:47:33.517210 5328 tablet_bootstrap.cc:654] T 2340240d7b034321b0cb4c0b35396d09 P 22bd06a8cbfb43519f365be45ade3a94: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:33.518245 5329 tablet_bootstrap.cc:654] T 2340240d7b034321b0cb4c0b35396d09 P fa05bdb779cb4da698075d717e9c2d39: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:33.518589 5328 log.cc:824] T 2340240d7b034321b0cb4c0b35396d09 P 22bd06a8cbfb43519f365be45ade3a94: Log is configured to *not* fsync() on all Append() calls
I20260621 07:47:33.519053 5329 log.cc:824] T 2340240d7b034321b0cb4c0b35396d09 P fa05bdb779cb4da698075d717e9c2d39: Log is configured to *not* fsync() on all Append() calls
I20260621 07:47:33.520532 5328 tablet_bootstrap.cc:492] T 2340240d7b034321b0cb4c0b35396d09 P 22bd06a8cbfb43519f365be45ade3a94: No bootstrap required, opened a new log
I20260621 07:47:33.520762 5329 tablet_bootstrap.cc:492] T 2340240d7b034321b0cb4c0b35396d09 P fa05bdb779cb4da698075d717e9c2d39: No bootstrap required, opened a new log
I20260621 07:47:33.520781 5328 ts_tablet_manager.cc:1397] T 2340240d7b034321b0cb4c0b35396d09 P 22bd06a8cbfb43519f365be45ade3a94: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260621 07:47:33.520921 5329 ts_tablet_manager.cc:1397] T 2340240d7b034321b0cb4c0b35396d09 P fa05bdb779cb4da698075d717e9c2d39: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260621 07:47:33.521690 5332 tablet_bootstrap.cc:492] T 2340240d7b034321b0cb4c0b35396d09 P c0c5bd10acb2484981118a3e42e56a11: Bootstrap starting.
I20260621 07:47:33.524037 5332 tablet_bootstrap.cc:654] T 2340240d7b034321b0cb4c0b35396d09 P c0c5bd10acb2484981118a3e42e56a11: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:33.524847 5332 log.cc:824] T 2340240d7b034321b0cb4c0b35396d09 P c0c5bd10acb2484981118a3e42e56a11: Log is configured to *not* fsync() on all Append() calls
I20260621 07:47:33.524955 5328 raft_consensus.cc:348] T 2340240d7b034321b0cb4c0b35396d09 P 22bd06a8cbfb43519f365be45ade3a94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa05bdb779cb4da698075d717e9c2d39" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 39067 } } peers { permanent_uuid: "22bd06a8cbfb43519f365be45ade3a94" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44939 } } peers { permanent_uuid: "c0c5bd10acb2484981118a3e42e56a11" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33089 } }
I20260621 07:47:33.525025 5329 raft_consensus.cc:348] T 2340240d7b034321b0cb4c0b35396d09 P fa05bdb779cb4da698075d717e9c2d39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa05bdb779cb4da698075d717e9c2d39" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 39067 } } peers { permanent_uuid: "22bd06a8cbfb43519f365be45ade3a94" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44939 } } peers { permanent_uuid: "c0c5bd10acb2484981118a3e42e56a11" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33089 } }
I20260621 07:47:33.525242 5329 raft_consensus.cc:374] T 2340240d7b034321b0cb4c0b35396d09 P fa05bdb779cb4da698075d717e9c2d39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:33.525242 5328 raft_consensus.cc:374] T 2340240d7b034321b0cb4c0b35396d09 P 22bd06a8cbfb43519f365be45ade3a94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:33.525296 5328 raft_consensus.cc:729] T 2340240d7b034321b0cb4c0b35396d09 P 22bd06a8cbfb43519f365be45ade3a94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22bd06a8cbfb43519f365be45ade3a94, State: Initialized, Role: FOLLOWER
I20260621 07:47:33.525324 5329 raft_consensus.cc:729] T 2340240d7b034321b0cb4c0b35396d09 P fa05bdb779cb4da698075d717e9c2d39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa05bdb779cb4da698075d717e9c2d39, State: Initialized, Role: FOLLOWER
I20260621 07:47:33.525744 5328 consensus_queue.cc:260] T 2340240d7b034321b0cb4c0b35396d09 P 22bd06a8cbfb43519f365be45ade3a94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa05bdb779cb4da698075d717e9c2d39" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 39067 } } peers { permanent_uuid: "22bd06a8cbfb43519f365be45ade3a94" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44939 } } peers { permanent_uuid: "c0c5bd10acb2484981118a3e42e56a11" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33089 } }
I20260621 07:47:33.525743 5329 consensus_queue.cc:260] T 2340240d7b034321b0cb4c0b35396d09 P fa05bdb779cb4da698075d717e9c2d39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa05bdb779cb4da698075d717e9c2d39" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 39067 } } peers { permanent_uuid: "22bd06a8cbfb43519f365be45ade3a94" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44939 } } peers { permanent_uuid: "c0c5bd10acb2484981118a3e42e56a11" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33089 } }
I20260621 07:47:33.526549 5332 tablet_bootstrap.cc:492] T 2340240d7b034321b0cb4c0b35396d09 P c0c5bd10acb2484981118a3e42e56a11: No bootstrap required, opened a new log
I20260621 07:47:33.526700 5329 ts_tablet_manager.cc:1428] T 2340240d7b034321b0cb4c0b35396d09 P fa05bdb779cb4da698075d717e9c2d39: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260621 07:47:33.526710 5328 ts_tablet_manager.cc:1428] T 2340240d7b034321b0cb4c0b35396d09 P 22bd06a8cbfb43519f365be45ade3a94: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260621 07:47:33.526733 5332 ts_tablet_manager.cc:1397] T 2340240d7b034321b0cb4c0b35396d09 P c0c5bd10acb2484981118a3e42e56a11: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260621 07:47:33.527295 5309 heartbeater.cc:503] Master 127.0.192.254:33209 was elected leader, sending a full tablet report...
I20260621 07:47:33.531216 5332 raft_consensus.cc:348] T 2340240d7b034321b0cb4c0b35396d09 P c0c5bd10acb2484981118a3e42e56a11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa05bdb779cb4da698075d717e9c2d39" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 39067 } } peers { permanent_uuid: "22bd06a8cbfb43519f365be45ade3a94" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44939 } } peers { permanent_uuid: "c0c5bd10acb2484981118a3e42e56a11" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33089 } }
I20260621 07:47:33.531503 5332 raft_consensus.cc:374] T 2340240d7b034321b0cb4c0b35396d09 P c0c5bd10acb2484981118a3e42e56a11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:33.531562 5332 raft_consensus.cc:729] T 2340240d7b034321b0cb4c0b35396d09 P c0c5bd10acb2484981118a3e42e56a11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0c5bd10acb2484981118a3e42e56a11, State: Initialized, Role: FOLLOWER
I20260621 07:47:33.531924 5332 consensus_queue.cc:260] T 2340240d7b034321b0cb4c0b35396d09 P c0c5bd10acb2484981118a3e42e56a11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa05bdb779cb4da698075d717e9c2d39" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 39067 } } peers { permanent_uuid: "22bd06a8cbfb43519f365be45ade3a94" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44939 } } peers { permanent_uuid: "c0c5bd10acb2484981118a3e42e56a11" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33089 } }
I20260621 07:47:33.532896 5332 ts_tablet_manager.cc:1428] T 2340240d7b034321b0cb4c0b35396d09 P c0c5bd10acb2484981118a3e42e56a11: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260621 07:47:33.585930 5335 raft_consensus.cc:482] T 2340240d7b034321b0cb4c0b35396d09 P 22bd06a8cbfb43519f365be45ade3a94 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:47:33.586156 5335 raft_consensus.cc:504] T 2340240d7b034321b0cb4c0b35396d09 P 22bd06a8cbfb43519f365be45ade3a94 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa05bdb779cb4da698075d717e9c2d39" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 39067 } } peers { permanent_uuid: "22bd06a8cbfb43519f365be45ade3a94" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44939 } } peers { permanent_uuid: "c0c5bd10acb2484981118a3e42e56a11" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33089 } }
I20260621 07:47:33.587260 5335 leader_election.cc:288] T 2340240d7b034321b0cb4c0b35396d09 P 22bd06a8cbfb43519f365be45ade3a94 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa05bdb779cb4da698075d717e9c2d39 (127.0.192.194:39067), c0c5bd10acb2484981118a3e42e56a11 (127.0.192.193:33089)
I20260621 07:47:33.592106 5327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:33.587933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:49475 (local address 127.0.192.193:33089)
0621 07:47:33.588052 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:33.588056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:33.588221 (+ 165us) server_negotiation.cc:407] Connection header received
0621 07:47:33.588304 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:33.588306 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:33.588367 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:33.588447 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:33.588695 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:33.589399 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:33.590784 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:33.591415 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:33.591539 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:33.591590 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:33.591903 (+ 313us) server_negotiation.cc:299] Negotiation successful
0621 07:47:33.591977 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 07:47:33.592106 5323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:33.587602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:42501 (local address 127.0.192.194:39067)
0621 07:47:33.587727 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:33.587731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:33.587760 (+ 29us) server_negotiation.cc:407] Connection header received
0621 07:47:33.587860 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:33.587863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:33.587912 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:33.587992 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:33.588389 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:33.589226 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:33.590784 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:33.591415 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:33.591540 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:33.591590 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:33.591903 (+ 313us) server_negotiation.cc:299] Negotiation successful
0621 07:47:33.591977 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260621 07:47:33.592141 5312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:33.587505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:39067 (local address 127.0.192.195:42501)
0621 07:47:33.587632 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:33.587649 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:33.587759 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:33.588100 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:33.588103 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:33.588122 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:33.588272 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:33.588282 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:33.589321 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:33.589323 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:33.590646 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:33.590657 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:33.591535 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:33.591538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:33.591745 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:33.591796 (+ 51us) client_negotiation.cc:769] Sending connection context
0621 07:47:33.591877 (+ 81us) client_negotiation.cc:240] Negotiation successful
0621 07:47:33.591976 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":164}
I20260621 07:47:33.592154 5337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:33.587802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:33089 (local address 127.0.192.195:49475)
0621 07:47:33.588123 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:33.588136 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:33.588227 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:33.588462 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:33.588464 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:33.588478 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:33.588606 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:33.588611 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:33.589492 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:33.589495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:33.590666 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:33.590672 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:33.591535 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:33.591538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:33.591742 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:33.591796 (+ 54us) client_negotiation.cc:769] Sending connection context
0621 07:47:33.591877 (+ 81us) client_negotiation.cc:240] Negotiation successful
0621 07:47:33.591976 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":72,"thread_start_us":140,"threads_started":1}
I20260621 07:47:33.592820 5129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2340240d7b034321b0cb4c0b35396d09" candidate_uuid: "22bd06a8cbfb43519f365be45ade3a94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa05bdb779cb4da698075d717e9c2d39" is_pre_election: true
I20260621 07:47:33.592816 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2340240d7b034321b0cb4c0b35396d09" candidate_uuid: "22bd06a8cbfb43519f365be45ade3a94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0c5bd10acb2484981118a3e42e56a11" is_pre_election: true
I20260621 07:47:33.593116 4995 raft_consensus.cc:2454] T 2340240d7b034321b0cb4c0b35396d09 P c0c5bd10acb2484981118a3e42e56a11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22bd06a8cbfb43519f365be45ade3a94 in term 0.
I20260621 07:47:33.593120 5129 raft_consensus.cc:2454] T 2340240d7b034321b0cb4c0b35396d09 P fa05bdb779cb4da698075d717e9c2d39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22bd06a8cbfb43519f365be45ade3a94 in term 0.
I20260621 07:47:33.593585 5199 leader_election.cc:302] T 2340240d7b034321b0cb4c0b35396d09 P 22bd06a8cbfb43519f365be45ade3a94 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22bd06a8cbfb43519f365be45ade3a94, c0c5bd10acb2484981118a3e42e56a11; no voters:
I20260621 07:47:33.593869 5335 raft_consensus.cc:2781] T 2340240d7b034321b0cb4c0b35396d09 P 22bd06a8cbfb43519f365be45ade3a94 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:47:33.593969 5335 raft_consensus.cc:482] T 2340240d7b034321b0cb4c0b35396d09 P 22bd06a8cbfb43519f365be45ade3a94 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:47:33.594030 5335 raft_consensus.cc:3037] T 2340240d7b034321b0cb4c0b35396d09 P 22bd06a8cbfb43519f365be45ade3a94 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:33.595329 5335 raft_consensus.cc:504] T 2340240d7b034321b0cb4c0b35396d09 P 22bd06a8cbfb43519f365be45ade3a94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa05bdb779cb4da698075d717e9c2d39" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 39067 } } peers { permanent_uuid: "22bd06a8cbfb43519f365be45ade3a94" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44939 } } peers { permanent_uuid: "c0c5bd10acb2484981118a3e42e56a11" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33089 } }
I20260621 07:47:33.595613 5335 leader_election.cc:288] T 2340240d7b034321b0cb4c0b35396d09 P 22bd06a8cbfb43519f365be45ade3a94 [CANDIDATE]: Term 1 election: Requested vote from peers fa05bdb779cb4da698075d717e9c2d39 (127.0.192.194:39067), c0c5bd10acb2484981118a3e42e56a11 (127.0.192.193:33089)
I20260621 07:47:33.595960 5129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2340240d7b034321b0cb4c0b35396d09" candidate_uuid: "22bd06a8cbfb43519f365be45ade3a94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa05bdb779cb4da698075d717e9c2d39"
I20260621 07:47:33.596015 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2340240d7b034321b0cb4c0b35396d09" candidate_uuid: "22bd06a8cbfb43519f365be45ade3a94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0c5bd10acb2484981118a3e42e56a11"
I20260621 07:47:33.596112 4995 raft_consensus.cc:3037] T 2340240d7b034321b0cb4c0b35396d09 P c0c5bd10acb2484981118a3e42e56a11 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:33.596094 5129 raft_consensus.cc:3037] T 2340240d7b034321b0cb4c0b35396d09 P fa05bdb779cb4da698075d717e9c2d39 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:33.597122 4995 raft_consensus.cc:2454] T 2340240d7b034321b0cb4c0b35396d09 P c0c5bd10acb2484981118a3e42e56a11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22bd06a8cbfb43519f365be45ade3a94 in term 1.
I20260621 07:47:33.597123 5129 raft_consensus.cc:2454] T 2340240d7b034321b0cb4c0b35396d09 P fa05bdb779cb4da698075d717e9c2d39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22bd06a8cbfb43519f365be45ade3a94 in term 1.
I20260621 07:47:33.597451 5199 leader_election.cc:302] T 2340240d7b034321b0cb4c0b35396d09 P 22bd06a8cbfb43519f365be45ade3a94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22bd06a8cbfb43519f365be45ade3a94, fa05bdb779cb4da698075d717e9c2d39; no voters:
I20260621 07:47:33.597656 5335 raft_consensus.cc:2781] T 2340240d7b034321b0cb4c0b35396d09 P 22bd06a8cbfb43519f365be45ade3a94 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:47:33.597898 5335 raft_consensus.cc:686] T 2340240d7b034321b0cb4c0b35396d09 P 22bd06a8cbfb43519f365be45ade3a94 [term 1 LEADER]: Becoming Leader. State: Replica: 22bd06a8cbfb43519f365be45ade3a94, State: Running, Role: LEADER
I20260621 07:47:33.598229 5335 consensus_queue.cc:237] T 2340240d7b034321b0cb4c0b35396d09 P 22bd06a8cbfb43519f365be45ade3a94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa05bdb779cb4da698075d717e9c2d39" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 39067 } } peers { permanent_uuid: "22bd06a8cbfb43519f365be45ade3a94" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44939 } } peers { permanent_uuid: "c0c5bd10acb2484981118a3e42e56a11" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33089 } }
I20260621 07:47:33.601119 4850 catalog_manager.cc:5495] T 2340240d7b034321b0cb4c0b35396d09 P 22bd06a8cbfb43519f365be45ade3a94 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22bd06a8cbfb43519f365be45ade3a94 (127.0.192.195). New cstate: current_term: 1 leader_uuid: "22bd06a8cbfb43519f365be45ade3a94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa05bdb779cb4da698075d717e9c2d39" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 39067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22bd06a8cbfb43519f365be45ade3a94" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0c5bd10acb2484981118a3e42e56a11" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33089 } health_report { overall_health: UNKNOWN } } }
I20260621 07:47:33.626200 5325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:33.622677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44260 (local address 127.0.192.195:44939)
0621 07:47:33.622808 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:33.622811 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:33.622826 (+ 15us) server_negotiation.cc:407] Connection header received
0621 07:47:33.622962 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:33.622964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:33.623018 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:33.623112 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:47:33.623323 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:33.623886 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:33.624820 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:33.625033 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:33.625149 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:33.625390 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:47:33.625780 (+ 390us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:47:33.625881 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:33.625938 (+ 57us) server_negotiation.cc:299] Negotiation successful
0621 07:47:33.625988 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 07:47:33.631623 4995 raft_consensus.cc:1261] T 2340240d7b034321b0cb4c0b35396d09 P c0c5bd10acb2484981118a3e42e56a11 [term 1 FOLLOWER]: Refusing update from remote peer 22bd06a8cbfb43519f365be45ade3a94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:47:33.631623 5129 raft_consensus.cc:1261] T 2340240d7b034321b0cb4c0b35396d09 P fa05bdb779cb4da698075d717e9c2d39 [term 1 FOLLOWER]: Refusing update from remote peer 22bd06a8cbfb43519f365be45ade3a94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:47:33.632295 5335 consensus_queue.cc:1035] T 2340240d7b034321b0cb4c0b35396d09 P 22bd06a8cbfb43519f365be45ade3a94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa05bdb779cb4da698075d717e9c2d39" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 39067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:47:33.632498 5338 consensus_queue.cc:1035] T 2340240d7b034321b0cb4c0b35396d09 P 22bd06a8cbfb43519f365be45ade3a94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0c5bd10acb2484981118a3e42e56a11" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:47:33.639885 5342 mvcc.cc:205] Tried to move back new op lower bound from 7299186907668848640 to 7299186907540942848. Current Snapshot: MvccSnapshot[applied={T|T < 7299186907668848640}]
I20260621 07:47:33.646744 4850 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:47:33.646868 4850 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:47:33.649133 4850 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f44e4c4fc10b4cf183b6b64b5fd42571: Sending DeleteTablet for 3 replicas of tablet 2340240d7b034321b0cb4c0b35396d09
I20260621 07:47:33.649791 5109 tablet_service.cc:1514] Processing DeleteTablet for tablet 2340240d7b034321b0cb4c0b35396d09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:47:33 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:47614
I20260621 07:47:33.649991 5243 tablet_service.cc:1514] Processing DeleteTablet for tablet 2340240d7b034321b0cb4c0b35396d09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:47:33 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:44248
I20260621 07:47:33.649988 4975 tablet_service.cc:1514] Processing DeleteTablet for tablet 2340240d7b034321b0cb4c0b35396d09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:47:33 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:60046
I20260621 07:47:33.650362 5351 tablet_replica.cc:330] T 2340240d7b034321b0cb4c0b35396d09 P fa05bdb779cb4da698075d717e9c2d39: stopping tablet replica
I20260621 07:47:33.650679 5351 raft_consensus.cc:2229] T 2340240d7b034321b0cb4c0b35396d09 P fa05bdb779cb4da698075d717e9c2d39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:47:33.650892 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 4913
I20260621 07:47:33.650980 5351 raft_consensus.cc:2258] T 2340240d7b034321b0cb4c0b35396d09 P fa05bdb779cb4da698075d717e9c2d39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:47:33.651480 5352 tablet_replica.cc:330] T 2340240d7b034321b0cb4c0b35396d09 P 22bd06a8cbfb43519f365be45ade3a94: stopping tablet replica
I20260621 07:47:33.651748 5352 raft_consensus.cc:2229] T 2340240d7b034321b0cb4c0b35396d09 P 22bd06a8cbfb43519f365be45ade3a94 [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:47:33.652058 5352 raft_consensus.cc:2258] T 2340240d7b034321b0cb4c0b35396d09 P 22bd06a8cbfb43519f365be45ade3a94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:47:33.653776 5352 ts_tablet_manager.cc:1905] T 2340240d7b034321b0cb4c0b35396d09 P 22bd06a8cbfb43519f365be45ade3a94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:47:33.655119 5351 ts_tablet_manager.cc:1905] T 2340240d7b034321b0cb4c0b35396d09 P fa05bdb779cb4da698075d717e9c2d39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:47:33.657214 5352 ts_tablet_manager.cc:1918] T 2340240d7b034321b0cb4c0b35396d09 P 22bd06a8cbfb43519f365be45ade3a94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:47:33.657297 5352 log.cc:1196] T 2340240d7b034321b0cb4c0b35396d09 P 22bd06a8cbfb43519f365be45ade3a94: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-2/wal/wals/2340240d7b034321b0cb4c0b35396d09
I20260621 07:47:33.657606 5352 ts_tablet_manager.cc:1939] T 2340240d7b034321b0cb4c0b35396d09 P 22bd06a8cbfb43519f365be45ade3a94: Deleting consensus metadata
I20260621 07:47:33.657716 5351 ts_tablet_manager.cc:1918] T 2340240d7b034321b0cb4c0b35396d09 P fa05bdb779cb4da698075d717e9c2d39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:47:33.657792 5351 log.cc:1196] T 2340240d7b034321b0cb4c0b35396d09 P fa05bdb779cb4da698075d717e9c2d39: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782028028522580-771-0/minicluster-data/ts-1/wal/wals/2340240d7b034321b0cb4c0b35396d09
I20260621 07:47:33.658066 5351 ts_tablet_manager.cc:1939] T 2340240d7b034321b0cb4c0b35396d09 P fa05bdb779cb4da698075d717e9c2d39: Deleting consensus metadata
I20260621 07:47:33.659256 4838 catalog_manager.cc:4841] TS 22bd06a8cbfb43519f365be45ade3a94 (127.0.192.195:44939): tablet 2340240d7b034321b0cb4c0b35396d09 (table test-table [id=ad3fbac0d9294532a2f03e66cafcb178]) successfully deleted
I20260621 07:47:33.659448 4838 catalog_manager.cc:4841] TS fa05bdb779cb4da698075d717e9c2d39 (127.0.192.194:39067): tablet 2340240d7b034321b0cb4c0b35396d09 (table test-table [id=ad3fbac0d9294532a2f03e66cafcb178]) successfully deleted
I20260621 07:47:33.659655 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 5047
W20260621 07:47:33.659633 4838 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.192.193:33089 (error 108)
I20260621 07:47:33.661952 5324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:33.661708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:33089 (local address 127.0.0.1:60058)
0621 07:47:33.661823 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:33.661888 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.193:33089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260621 07:47:33.662120 4838 catalog_manager.cc:4584] TS c0c5bd10acb2484981118a3e42e56a11 (127.0.192.193:33089): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2340240d7b034321b0cb4c0b35396d09: Network error: Client connection negotiation failed: client connection to 127.0.192.193:33089: connect: Connection refused (error 111)
I20260621 07:47:33.666432 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 5181
I20260621 07:47:33.671368 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 4820
2026-06-21T07:47:33Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (5718 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-21T07:47:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:47:33Z Disabled control of system clock
I20260621 07:47:33.700784 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:46129
--webserver_interface=127.0.192.254
--webserver_port=0
--builtin_ntp_servers=127.0.192.212:34201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:46129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260621 07:47:33.798319 5358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:33.798524 5358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:33.798555 5358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:33.801893 5358 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:47:33.801935 5358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:33.801949 5358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:33.801964 5358 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:47:33.801977 5358 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:47:33.806234 5358 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:34201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:46129
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:46129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:33.807045 5358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:33.808097 5358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:33.814970 5363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:33.815061 5366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:33.815011 5364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:33.815267 5358 server_base.cc:1034] running on GCE node
I20260621 07:47:33.815722 5358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:33.816725 5358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:33.817873 5358 hybrid_clock.cc:648] HybridClock initialized: now 1782028053817857 us; error 28 us; skew 500 ppm
I20260621 07:47:33.819417 5358 webserver.cc:460] Webserver started at http://127.0.192.254:32999/ using document root <none> and password file <none>
I20260621 07:47:33.819906 5358 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:33.819962 5358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:33.820160 5358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:33.821911 5358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "52706a9c0cc14e40b375840876d6adce"
format_stamp: "Formatted at 2026-06-21 07:47:33 on dist-test-slave-lzx3"
server_key: "ee6b8704d96cb152772db76ddfa40b07"
server_key_iv: "3c8ca8be7d6caaa352fcf50544fd42e9"
server_key_version: "encryptionkey@0"
I20260621 07:47:33.822441 5358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "52706a9c0cc14e40b375840876d6adce"
format_stamp: "Formatted at 2026-06-21 07:47:33 on dist-test-slave-lzx3"
server_key: "ee6b8704d96cb152772db76ddfa40b07"
server_key_iv: "3c8ca8be7d6caaa352fcf50544fd42e9"
server_key_version: "encryptionkey@0"
I20260621 07:47:33.825695 5358 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260621 07:47:33.827836 5372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:33.828919 5358 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:33.829035 5358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "52706a9c0cc14e40b375840876d6adce"
format_stamp: "Formatted at 2026-06-21 07:47:33 on dist-test-slave-lzx3"
server_key: "ee6b8704d96cb152772db76ddfa40b07"
server_key_iv: "3c8ca8be7d6caaa352fcf50544fd42e9"
server_key_version: "encryptionkey@0"
I20260621 07:47:33.829123 5358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:33.833071 5358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:33.833572 5358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:33.833734 5358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:33.839782 5358 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:46129
I20260621 07:47:33.839789 5424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:46129 every 8 connection(s)
I20260621 07:47:33.840718 5358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:47:33.843276 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:33.845178 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 5358
I20260621 07:47:33.845300 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:47:33.845588 771 external_mini_cluster.cc:1428] Setting key c441ad2ef3469b785d079d47f58e212d
I20260621 07:47:33.848546 5425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52706a9c0cc14e40b375840876d6adce: Bootstrap starting.
I20260621 07:47:33.850453 5428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:33.846858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37178 (local address 127.0.192.254:46129)
0621 07:47:33.847438 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:33.847450 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:33.847478 (+ 28us) server_negotiation.cc:407] Connection header received
0621 07:47:33.848034 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:33.848048 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:33.848376 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:33.848738 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:33.849399 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:33.849403 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:33.849413 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:47:33.849422 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:33.849470 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:47:33.849483 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:33.849611 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:33.849675 (+ 64us) server_negotiation.cc:299] Negotiation successful
0621 07:47:33.849893 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":124,"threads_started":1}
I20260621 07:47:33.850638 5425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52706a9c0cc14e40b375840876d6adce: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:33.851291 5425 log.cc:824] T 00000000000000000000000000000000 P 52706a9c0cc14e40b375840876d6adce: Log is configured to *not* fsync() on all Append() calls
I20260621 07:47:33.852998 5425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52706a9c0cc14e40b375840876d6adce: No bootstrap required, opened a new log
I20260621 07:47:33.856230 5425 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52706a9c0cc14e40b375840876d6adce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52706a9c0cc14e40b375840876d6adce" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 46129 } }
I20260621 07:47:33.856410 5425 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52706a9c0cc14e40b375840876d6adce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:33.856446 5425 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52706a9c0cc14e40b375840876d6adce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52706a9c0cc14e40b375840876d6adce, State: Initialized, Role: FOLLOWER
I20260621 07:47:33.856853 5425 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52706a9c0cc14e40b375840876d6adce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52706a9c0cc14e40b375840876d6adce" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 46129 } }
I20260621 07:47:33.856979 5430 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52706a9c0cc14e40b375840876d6adce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:47:33.857125 5430 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52706a9c0cc14e40b375840876d6adce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52706a9c0cc14e40b375840876d6adce" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 46129 } }
I20260621 07:47:33.857426 5425 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52706a9c0cc14e40b375840876d6adce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:47:33.857425 5430 leader_election.cc:302] T 00000000000000000000000000000000 P 52706a9c0cc14e40b375840876d6adce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52706a9c0cc14e40b375840876d6adce; no voters:
I20260621 07:47:33.857515 5425 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52706a9c0cc14e40b375840876d6adce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:47:33.857568 5425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52706a9c0cc14e40b375840876d6adce [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:33.857743 5430 leader_election.cc:288] T 00000000000000000000000000000000 P 52706a9c0cc14e40b375840876d6adce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260621 07:47:33.858276 5425 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52706a9c0cc14e40b375840876d6adce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52706a9c0cc14e40b375840876d6adce" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 46129 } }
I20260621 07:47:33.858384 5425 leader_election.cc:302] T 00000000000000000000000000000000 P 52706a9c0cc14e40b375840876d6adce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52706a9c0cc14e40b375840876d6adce; no voters:
I20260621 07:47:33.858418 5431 raft_consensus.cc:2741] T 00000000000000000000000000000000 P 52706a9c0cc14e40b375840876d6adce [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260621 07:47:33.858469 5425 leader_election.cc:288] T 00000000000000000000000000000000 P 52706a9c0cc14e40b375840876d6adce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:47:33.858618 5431 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52706a9c0cc14e40b375840876d6adce [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:47:33.858685 5431 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52706a9c0cc14e40b375840876d6adce [term 1 LEADER]: Becoming Leader. State: Replica: 52706a9c0cc14e40b375840876d6adce, State: Running, Role: LEADER
I20260621 07:47:33.858964 5431 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52706a9c0cc14e40b375840876d6adce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52706a9c0cc14e40b375840876d6adce" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 46129 } }
I20260621 07:47:33.859441 5425 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52706a9c0cc14e40b375840876d6adce [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:47:33.859952 5431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52706a9c0cc14e40b375840876d6adce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52706a9c0cc14e40b375840876d6adce. Latest consensus state: current_term: 1 leader_uuid: "52706a9c0cc14e40b375840876d6adce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52706a9c0cc14e40b375840876d6adce" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 46129 } } }
I20260621 07:47:33.860124 5431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52706a9c0cc14e40b375840876d6adce [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:33.860262 5432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52706a9c0cc14e40b375840876d6adce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52706a9c0cc14e40b375840876d6adce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52706a9c0cc14e40b375840876d6adce" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 46129 } } }
I20260621 07:47:33.860430 5432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52706a9c0cc14e40b375840876d6adce [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:33.860929 5439 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:47:33.863601 5439 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:47:33.868450 5439 catalog_manager.cc:1347] Generated new cluster ID: b846d2f57998421fa1dceacf06d52c70
I20260621 07:47:33.868505 5439 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:47:33.888273 5439 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:47:33.888541 5439 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:47:33.896936 5439 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52706a9c0cc14e40b375840876d6adce: Generated new TSK 0
I20260621 07:47:33.897565 5439 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:47:33.905659 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.193:0
--local_ip_for_outbound_sockets=127.0.192.193
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:46129
--builtin_ntp_servers=127.0.192.212:34201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260621 07:47:34.004349 5449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:34.004607 5449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:34.004690 5449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:34.008188 5449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:34.008250 5449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:34.008319 5449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.193
I20260621 07:47:34.012678 5449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:34201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:46129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.193
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:34.013541 5449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:34.014801 5449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:34.023655 5457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:34.023705 5455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:34.023784 5449 server_base.cc:1034] running on GCE node
W20260621 07:47:34.023854 5454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:34.024279 5449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:34.024976 5449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:34.026139 5449 hybrid_clock.cc:648] HybridClock initialized: now 1782028054026109 us; error 41 us; skew 500 ppm
I20260621 07:47:34.028153 5449 webserver.cc:460] Webserver started at http://127.0.192.193:44811/ using document root <none> and password file <none>
I20260621 07:47:34.028795 5449 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:34.028879 5449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:34.029105 5449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:34.031097 5449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-0/data/instance:
uuid: "6b5db6fd4e9449668a3e23baa7966788"
format_stamp: "Formatted at 2026-06-21 07:47:34 on dist-test-slave-lzx3"
server_key: "bf914a3b4fad70311f1ea20750728e65"
server_key_iv: "c569dc6e07a4b0e986166e0cf49c6c90"
server_key_version: "encryptionkey@0"
I20260621 07:47:34.031637 5449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-0/wal/instance:
uuid: "6b5db6fd4e9449668a3e23baa7966788"
format_stamp: "Formatted at 2026-06-21 07:47:34 on dist-test-slave-lzx3"
server_key: "bf914a3b4fad70311f1ea20750728e65"
server_key_iv: "c569dc6e07a4b0e986166e0cf49c6c90"
server_key_version: "encryptionkey@0"
I20260621 07:47:34.035353 5449 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260621 07:47:34.037833 5463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:34.038964 5449 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 07:47:34.039113 5449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-0/wal
uuid: "6b5db6fd4e9449668a3e23baa7966788"
format_stamp: "Formatted at 2026-06-21 07:47:34 on dist-test-slave-lzx3"
server_key: "bf914a3b4fad70311f1ea20750728e65"
server_key_iv: "c569dc6e07a4b0e986166e0cf49c6c90"
server_key_version: "encryptionkey@0"
I20260621 07:47:34.039234 5449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:34.043733 5449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:34.044307 5449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:34.044495 5449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:34.045040 5449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:47:34.045883 5449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:47:34.045955 5449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:34.046020 5449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:47:34.046077 5449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:34.054688 5449 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.193:39849
I20260621 07:47:34.054711 5576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.193:39849 every 8 connection(s)
I20260621 07:47:34.055541 5449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
I20260621 07:47:34.061127 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 5449
I20260621 07:47:34.061251 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-0/wal/instance
I20260621 07:47:34.061388 5428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:34.057290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:41383 (local address 127.0.192.254:46129)
0621 07:47:34.057430 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:34.057434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:34.058330 (+ 896us) server_negotiation.cc:407] Connection header received
0621 07:47:34.059692 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:34.059696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:34.059739 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:34.059839 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:34.060922 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:34.060926 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:34.060929 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:47:34.060940 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:34.060962 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:47:34.060971 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:34.061075 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:34.061198 (+ 123us) server_negotiation.cc:299] Negotiation successful
0621 07:47:34.061240 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 07:47:34.061619 771 external_mini_cluster.cc:1428] Setting key 95bb601165875a1b3534882d7a58a44f
I20260621 07:47:34.062294 5579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:34.057624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:46129 (local address 127.0.192.193:41383)
0621 07:47:34.058192 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:34.058227 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:34.059431 (+ 1204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:34.059990 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:34.059997 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:34.060094 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:34.060694 (+ 600us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:47:34.060696 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:34.060730 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:47:34.060789 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:34.061061 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:34.061069 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:34.061074 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 07:47:34.061236 (+ 162us) client_negotiation.cc:240] Negotiation successful
0621 07:47:34.061447 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":109,"threads_started":1}
I20260621 07:47:34.063373 5577 heartbeater.cc:347] Connected to a master server at 127.0.192.254:46129
I20260621 07:47:34.063475 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.194:0
--local_ip_for_outbound_sockets=127.0.192.194
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:46129
--builtin_ntp_servers=127.0.192.212:34201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260621 07:47:34.063598 5577 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:34.063956 5577 heartbeater.cc:511] Master 127.0.192.254:46129 requested a full tablet report, sending...
I20260621 07:47:34.065274 5389 ts_manager.cc:194] Registered new tserver with Master: 6b5db6fd4e9449668a3e23baa7966788 (127.0.192.193:39849)
W20260621 07:47:34.168125 5580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:34.168349 5580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:34.168388 5580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:34.172086 5580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:34.172137 5580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:34.172252 5580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.194
I20260621 07:47:34.176739 5580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:34201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:46129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.194
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:34.177618 5580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:34.178881 5580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 07:47:34.187836 5580 server_base.cc:1034] running on GCE node
W20260621 07:47:34.187793 5585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:34.187848 5588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:34.187788 5586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:34.188473 5580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:34.189038 5580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:34.190203 5580 hybrid_clock.cc:648] HybridClock initialized: now 1782028054190189 us; error 26 us; skew 500 ppm
I20260621 07:47:34.191673 5580 webserver.cc:460] Webserver started at http://127.0.192.194:46399/ using document root <none> and password file <none>
I20260621 07:47:34.192246 5580 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:34.192335 5580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:34.192538 5580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:34.194409 5580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-1/data/instance:
uuid: "d779c413657b43cd8237b6dfd61ac2e9"
format_stamp: "Formatted at 2026-06-21 07:47:34 on dist-test-slave-lzx3"
server_key: "daaa53ffd930dca6ba2031ad710ead6a"
server_key_iv: "2d9f0d20f38b66db6b682873dbbfae98"
server_key_version: "encryptionkey@0"
I20260621 07:47:34.194944 5580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-1/wal/instance:
uuid: "d779c413657b43cd8237b6dfd61ac2e9"
format_stamp: "Formatted at 2026-06-21 07:47:34 on dist-test-slave-lzx3"
server_key: "daaa53ffd930dca6ba2031ad710ead6a"
server_key_iv: "2d9f0d20f38b66db6b682873dbbfae98"
server_key_version: "encryptionkey@0"
I20260621 07:47:34.198216 5580 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 07:47:34.200205 5594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:34.201344 5580 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:34.201452 5580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-1/wal
uuid: "d779c413657b43cd8237b6dfd61ac2e9"
format_stamp: "Formatted at 2026-06-21 07:47:34 on dist-test-slave-lzx3"
server_key: "daaa53ffd930dca6ba2031ad710ead6a"
server_key_iv: "2d9f0d20f38b66db6b682873dbbfae98"
server_key_version: "encryptionkey@0"
I20260621 07:47:34.201557 5580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:34.205375 5580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:34.205825 5580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:34.205940 5580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:34.206382 5580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:47:34.207123 5580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:47:34.207166 5580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:34.207201 5580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:47:34.207223 5580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:34.215291 5580 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.194:35435
I20260621 07:47:34.215310 5707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.194:35435 every 8 connection(s)
I20260621 07:47:34.216042 5580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
I20260621 07:47:34.218725 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 5580
I20260621 07:47:34.218863 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-1/wal/instance
I20260621 07:47:34.219110 771 external_mini_cluster.cc:1428] Setting key f08079d5f31af68c900a1b875b248740
I20260621 07:47:34.221289 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.195:0
--local_ip_for_outbound_sockets=127.0.192.195
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:46129
--builtin_ntp_servers=127.0.192.212:34201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260621 07:47:34.222098 5428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:34.217639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:58459 (local address 127.0.192.254:46129)
0621 07:47:34.217779 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:34.217785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:34.218645 (+ 860us) server_negotiation.cc:407] Connection header received
0621 07:47:34.220194 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:34.220196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:34.220229 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:34.220299 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:34.221589 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:34.221595 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:34.221601 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:47:34.221619 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:34.221642 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:47:34.221654 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:34.221744 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:34.221880 (+ 136us) server_negotiation.cc:299] Negotiation successful
0621 07:47:34.221924 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260621 07:47:34.222822 5710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:34.217910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:46129 (local address 127.0.192.194:58459)
0621 07:47:34.218486 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:34.218543 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:34.219961 (+ 1418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:34.220477 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:34.220486 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:34.220621 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:34.221241 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:47:34.221242 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:34.221277 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:47:34.221338 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:34.221755 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:34.221761 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:34.221765 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 07:47:34.221887 (+ 122us) client_negotiation.cc:240] Negotiation successful
0621 07:47:34.222049 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":99,"threads_started":1}
I20260621 07:47:34.223869 5708 heartbeater.cc:347] Connected to a master server at 127.0.192.254:46129
I20260621 07:47:34.224074 5708 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:34.224416 5708 heartbeater.cc:511] Master 127.0.192.254:46129 requested a full tablet report, sending...
I20260621 07:47:34.225368 5389 ts_manager.cc:194] Registered new tserver with Master: d779c413657b43cd8237b6dfd61ac2e9 (127.0.192.194:35435)
W20260621 07:47:34.324834 5711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:34.325042 5711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:34.325076 5711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:34.328852 5711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:34.328894 5711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:34.328940 5711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.195
I20260621 07:47:34.333696 5711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:34201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:46129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.195
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:34.334523 5711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:34.335691 5711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:34.344360 5716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:34.344452 5717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:34.344610 5711 server_base.cc:1034] running on GCE node
W20260621 07:47:34.344409 5719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:34.345081 5711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:34.345564 5711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:34.346709 5711 hybrid_clock.cc:648] HybridClock initialized: now 1782028054346694 us; error 30 us; skew 500 ppm
I20260621 07:47:34.348367 5711 webserver.cc:460] Webserver started at http://127.0.192.195:38437/ using document root <none> and password file <none>
I20260621 07:47:34.348924 5711 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:34.348985 5711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:34.349218 5711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:34.350922 5711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-2/data/instance:
uuid: "69c1334142ef4ff2af2cff93bfaccce7"
format_stamp: "Formatted at 2026-06-21 07:47:34 on dist-test-slave-lzx3"
server_key: "cdeea14d34f5131843f7af4d6e4e9c13"
server_key_iv: "fa631112f45ef755cd9b4e3e9a9cfedb"
server_key_version: "encryptionkey@0"
I20260621 07:47:34.351404 5711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-2/wal/instance:
uuid: "69c1334142ef4ff2af2cff93bfaccce7"
format_stamp: "Formatted at 2026-06-21 07:47:34 on dist-test-slave-lzx3"
server_key: "cdeea14d34f5131843f7af4d6e4e9c13"
server_key_iv: "fa631112f45ef755cd9b4e3e9a9cfedb"
server_key_version: "encryptionkey@0"
I20260621 07:47:34.354748 5711 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260621 07:47:34.357044 5725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:34.358095 5711 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260621 07:47:34.358210 5711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-2/wal
uuid: "69c1334142ef4ff2af2cff93bfaccce7"
format_stamp: "Formatted at 2026-06-21 07:47:34 on dist-test-slave-lzx3"
server_key: "cdeea14d34f5131843f7af4d6e4e9c13"
server_key_iv: "fa631112f45ef755cd9b4e3e9a9cfedb"
server_key_version: "encryptionkey@0"
I20260621 07:47:34.358299 5711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:34.363483 5711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:34.363988 5711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:34.364140 5711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:34.364701 5711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:47:34.365492 5711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:47:34.365556 5711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:34.365619 5711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:47:34.365665 5711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:34.373713 5711 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.195:33097
I20260621 07:47:34.373767 5838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.195:33097 every 8 connection(s)
I20260621 07:47:34.374531 5711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
I20260621 07:47:34.376897 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 5711
I20260621 07:47:34.377017 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-2/wal/instance
I20260621 07:47:34.377286 771 external_mini_cluster.cc:1428] Setting key e7c48b671edf393269dd85674464b639
I20260621 07:47:34.379966 5428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:34.376022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:57121 (local address 127.0.192.254:46129)
0621 07:47:34.376134 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:34.376138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:34.377173 (+ 1035us) server_negotiation.cc:407] Connection header received
0621 07:47:34.378624 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:34.378626 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:34.378666 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:34.378714 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:34.379613 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:34.379615 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:34.379617 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:47:34.379626 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:34.379659 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:47:34.379666 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:34.379727 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:34.379837 (+ 110us) server_negotiation.cc:299] Negotiation successful
0621 07:47:34.379868 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260621 07:47:34.380800 5841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:34.376389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:46129 (local address 127.0.192.195:57121)
0621 07:47:34.377010 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:34.377075 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:34.378450 (+ 1375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:34.378863 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:34.378885 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:34.378979 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:34.379429 (+ 450us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:47:34.379431 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:34.379460 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:47:34.379514 (+ 54us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:34.379727 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:34.379732 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:34.379735 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 07:47:34.379872 (+ 137us) client_negotiation.cc:240] Negotiation successful
0621 07:47:34.380088 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":138,"threads_started":1}
I20260621 07:47:34.381716 5839 heartbeater.cc:347] Connected to a master server at 127.0.192.254:46129
I20260621 07:47:34.381891 5839 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:34.382193 5839 heartbeater.cc:511] Master 127.0.192.254:46129 requested a full tablet report, sending...
I20260621 07:47:34.383005 5389 ts_manager.cc:194] Registered new tserver with Master: 69c1334142ef4ff2af2cff93bfaccce7 (127.0.192.195:33097)
I20260621 07:47:34.390383 771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:47:34.393363 5428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:34.391974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37184 (local address 127.0.192.254:46129)
0621 07:47:34.392092 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:34.392095 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:34.392257 (+ 162us) server_negotiation.cc:407] Connection header received
0621 07:47:34.392347 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:34.392349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:34.392374 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:34.392430 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:34.393032 (+ 602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:34.393034 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:34.393036 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:47:34.393046 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:34.393061 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:47:34.393068 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:34.393134 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:34.393199 (+ 65us) server_negotiation.cc:299] Negotiation successful
0621 07:47:34.393226 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260621 07:47:34.395694 5389 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37184:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:47:34.398669 5389 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:47:34.408115 5850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:34.405344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.195:33097 (local address 127.0.0.1:34224)
0621 07:47:34.405647 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:34.405735 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:34.405828 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:34.406838 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:34.406843 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:34.406907 (+ 64us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:34.407400 (+ 493us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:47:34.407404 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:34.407425 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:47:34.407481 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:34.407791 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:34.407798 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:34.407803 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 07:47:34.407903 (+ 100us) client_negotiation.cc:240] Negotiation successful
0621 07:47:34.407959 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":75,"threads_started":1}
I20260621 07:47:34.408128 5851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:34.405453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34224 (local address 127.0.192.195:33097)
0621 07:47:34.405794 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:34.405804 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:34.405825 (+ 21us) server_negotiation.cc:407] Connection header received
0621 07:47:34.405910 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:34.405914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:34.406075 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:34.406221 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:34.407606 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:34.407612 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:34.407623 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:47:34.407634 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:34.407674 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:47:34.407701 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:34.407819 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:34.407914 (+ 95us) server_negotiation.cc:299] Negotiation successful
0621 07:47:34.407975 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":102,"threads_started":1}
I20260621 07:47:34.408820 5852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:34.405949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:35435 (local address 127.0.0.1:56342)
0621 07:47:34.406474 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:34.406490 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:34.406577 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:34.407045 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:34.407050 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:34.407066 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:34.407622 (+ 556us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:47:34.407625 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:34.407634 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:47:34.407655 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:34.408572 (+ 917us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:34.408592 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:34.408596 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 07:47:34.408669 (+ 73us) client_negotiation.cc:240] Negotiation successful
0621 07:47:34.408701 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":69,"threads_started":1}
I20260621 07:47:34.409204 5853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:34.406084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56342 (local address 127.0.192.194:35435)
0621 07:47:34.406498 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:34.406510 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:34.406666 (+ 156us) server_negotiation.cc:407] Connection header received
0621 07:47:34.406721 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:34.406726 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:34.406909 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:34.407065 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:34.408098 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:34.408103 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:34.408112 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:47:34.408123 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:34.408157 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:47:34.408428 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:34.408527 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:34.408986 (+ 459us) server_negotiation.cc:299] Negotiation successful
0621 07:47:34.409054 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":126,"threads_started":1}
I20260621 07:47:34.409358 5854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:34.406262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:39849 (local address 127.0.0.1:58792)
0621 07:47:34.407240 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:34.407254 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:34.407344 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:34.408359 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:34.408361 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:34.408376 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:34.408767 (+ 391us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:47:34.408769 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:34.408778 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:47:34.408806 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:34.409138 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:34.409141 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:34.409145 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 07:47:34.409214 (+ 69us) client_negotiation.cc:240] Negotiation successful
0621 07:47:34.409248 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":904,"thread_start_us":87,"threads_started":1}
I20260621 07:47:34.409715 5855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:34.406741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58792 (local address 127.0.192.193:39849)
0621 07:47:34.407177 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:34.407191 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:34.407349 (+ 158us) server_negotiation.cc:407] Connection header received
0621 07:47:34.407444 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:34.407450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:34.407620 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:34.407746 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:34.408929 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:34.408933 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:34.408942 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:47:34.408953 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:34.408990 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:47:34.409014 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:34.409100 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:34.409487 (+ 387us) server_negotiation.cc:299] Negotiation successful
0621 07:47:34.409552 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":146,"threads_started":1}
I20260621 07:47:34.410928 5773 tablet_service.cc:1467] Processing CreateTablet for tablet b97fc3d6dc734b049ecc0ade547ed01a (DEFAULT_TABLE table=test-table [id=e4125d58ae1f443da639ae79dd55109e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:47:34.411239 5642 tablet_service.cc:1467] Processing CreateTablet for tablet b97fc3d6dc734b049ecc0ade547ed01a (DEFAULT_TABLE table=test-table [id=e4125d58ae1f443da639ae79dd55109e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:47:34.411870 5773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b97fc3d6dc734b049ecc0ade547ed01a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:34.412060 5642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b97fc3d6dc734b049ecc0ade547ed01a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:34.411919 5511 tablet_service.cc:1467] Processing CreateTablet for tablet b97fc3d6dc734b049ecc0ade547ed01a (DEFAULT_TABLE table=test-table [id=e4125d58ae1f443da639ae79dd55109e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:47:34.412926 5511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b97fc3d6dc734b049ecc0ade547ed01a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:34.417512 5856 tablet_bootstrap.cc:492] T b97fc3d6dc734b049ecc0ade547ed01a P d779c413657b43cd8237b6dfd61ac2e9: Bootstrap starting.
I20260621 07:47:34.418771 5857 tablet_bootstrap.cc:492] T b97fc3d6dc734b049ecc0ade547ed01a P 69c1334142ef4ff2af2cff93bfaccce7: Bootstrap starting.
I20260621 07:47:34.419402 5856 tablet_bootstrap.cc:654] T b97fc3d6dc734b049ecc0ade547ed01a P d779c413657b43cd8237b6dfd61ac2e9: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:34.420023 5856 log.cc:824] T b97fc3d6dc734b049ecc0ade547ed01a P d779c413657b43cd8237b6dfd61ac2e9: Log is configured to *not* fsync() on all Append() calls
I20260621 07:47:34.420406 5857 tablet_bootstrap.cc:654] T b97fc3d6dc734b049ecc0ade547ed01a P 69c1334142ef4ff2af2cff93bfaccce7: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:34.420814 5858 tablet_bootstrap.cc:492] T b97fc3d6dc734b049ecc0ade547ed01a P 6b5db6fd4e9449668a3e23baa7966788: Bootstrap starting.
I20260621 07:47:34.421173 5857 log.cc:824] T b97fc3d6dc734b049ecc0ade547ed01a P 69c1334142ef4ff2af2cff93bfaccce7: Log is configured to *not* fsync() on all Append() calls
I20260621 07:47:34.421831 5856 tablet_bootstrap.cc:492] T b97fc3d6dc734b049ecc0ade547ed01a P d779c413657b43cd8237b6dfd61ac2e9: No bootstrap required, opened a new log
I20260621 07:47:34.422019 5856 ts_tablet_manager.cc:1397] T b97fc3d6dc734b049ecc0ade547ed01a P d779c413657b43cd8237b6dfd61ac2e9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260621 07:47:34.422859 5858 tablet_bootstrap.cc:654] T b97fc3d6dc734b049ecc0ade547ed01a P 6b5db6fd4e9449668a3e23baa7966788: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:34.423035 5857 tablet_bootstrap.cc:492] T b97fc3d6dc734b049ecc0ade547ed01a P 69c1334142ef4ff2af2cff93bfaccce7: No bootstrap required, opened a new log
I20260621 07:47:34.423202 5857 ts_tablet_manager.cc:1397] T b97fc3d6dc734b049ecc0ade547ed01a P 69c1334142ef4ff2af2cff93bfaccce7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260621 07:47:34.423693 5858 log.cc:824] T b97fc3d6dc734b049ecc0ade547ed01a P 6b5db6fd4e9449668a3e23baa7966788: Log is configured to *not* fsync() on all Append() calls
I20260621 07:47:34.425130 5858 tablet_bootstrap.cc:492] T b97fc3d6dc734b049ecc0ade547ed01a P 6b5db6fd4e9449668a3e23baa7966788: No bootstrap required, opened a new log
I20260621 07:47:34.425318 5858 ts_tablet_manager.cc:1397] T b97fc3d6dc734b049ecc0ade547ed01a P 6b5db6fd4e9449668a3e23baa7966788: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260621 07:47:34.426286 5856 raft_consensus.cc:348] T b97fc3d6dc734b049ecc0ade547ed01a P d779c413657b43cd8237b6dfd61ac2e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69c1334142ef4ff2af2cff93bfaccce7" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 33097 } } peers { permanent_uuid: "d779c413657b43cd8237b6dfd61ac2e9" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 35435 } } peers { permanent_uuid: "6b5db6fd4e9449668a3e23baa7966788" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 39849 } }
I20260621 07:47:34.426617 5856 raft_consensus.cc:374] T b97fc3d6dc734b049ecc0ade547ed01a P d779c413657b43cd8237b6dfd61ac2e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:34.426715 5856 raft_consensus.cc:729] T b97fc3d6dc734b049ecc0ade547ed01a P d779c413657b43cd8237b6dfd61ac2e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d779c413657b43cd8237b6dfd61ac2e9, State: Initialized, Role: FOLLOWER
I20260621 07:47:34.427201 5856 consensus_queue.cc:260] T b97fc3d6dc734b049ecc0ade547ed01a P d779c413657b43cd8237b6dfd61ac2e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69c1334142ef4ff2af2cff93bfaccce7" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 33097 } } peers { permanent_uuid: "d779c413657b43cd8237b6dfd61ac2e9" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 35435 } } peers { permanent_uuid: "6b5db6fd4e9449668a3e23baa7966788" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 39849 } }
I20260621 07:47:34.427296 5857 raft_consensus.cc:348] T b97fc3d6dc734b049ecc0ade547ed01a P 69c1334142ef4ff2af2cff93bfaccce7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69c1334142ef4ff2af2cff93bfaccce7" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 33097 } } peers { permanent_uuid: "d779c413657b43cd8237b6dfd61ac2e9" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 35435 } } peers { permanent_uuid: "6b5db6fd4e9449668a3e23baa7966788" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 39849 } }
I20260621 07:47:34.427541 5857 raft_consensus.cc:374] T b97fc3d6dc734b049ecc0ade547ed01a P 69c1334142ef4ff2af2cff93bfaccce7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:34.427640 5857 raft_consensus.cc:729] T b97fc3d6dc734b049ecc0ade547ed01a P 69c1334142ef4ff2af2cff93bfaccce7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69c1334142ef4ff2af2cff93bfaccce7, State: Initialized, Role: FOLLOWER
I20260621 07:47:34.428020 5857 consensus_queue.cc:260] T b97fc3d6dc734b049ecc0ade547ed01a P 69c1334142ef4ff2af2cff93bfaccce7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69c1334142ef4ff2af2cff93bfaccce7" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 33097 } } peers { permanent_uuid: "d779c413657b43cd8237b6dfd61ac2e9" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 35435 } } peers { permanent_uuid: "6b5db6fd4e9449668a3e23baa7966788" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 39849 } }
I20260621 07:47:34.428196 5856 ts_tablet_manager.cc:1428] T b97fc3d6dc734b049ecc0ade547ed01a P d779c413657b43cd8237b6dfd61ac2e9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260621 07:47:34.428495 5708 heartbeater.cc:503] Master 127.0.192.254:46129 was elected leader, sending a full tablet report...
I20260621 07:47:34.428861 5857 ts_tablet_manager.cc:1428] T b97fc3d6dc734b049ecc0ade547ed01a P 69c1334142ef4ff2af2cff93bfaccce7: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260621 07:47:34.428946 5858 raft_consensus.cc:348] T b97fc3d6dc734b049ecc0ade547ed01a P 6b5db6fd4e9449668a3e23baa7966788 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69c1334142ef4ff2af2cff93bfaccce7" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 33097 } } peers { permanent_uuid: "d779c413657b43cd8237b6dfd61ac2e9" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 35435 } } peers { permanent_uuid: "6b5db6fd4e9449668a3e23baa7966788" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 39849 } }
I20260621 07:47:34.429147 5839 heartbeater.cc:503] Master 127.0.192.254:46129 was elected leader, sending a full tablet report...
I20260621 07:47:34.429225 5858 raft_consensus.cc:374] T b97fc3d6dc734b049ecc0ade547ed01a P 6b5db6fd4e9449668a3e23baa7966788 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:34.429278 5858 raft_consensus.cc:729] T b97fc3d6dc734b049ecc0ade547ed01a P 6b5db6fd4e9449668a3e23baa7966788 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b5db6fd4e9449668a3e23baa7966788, State: Initialized, Role: FOLLOWER
I20260621 07:47:34.429651 5858 consensus_queue.cc:260] T b97fc3d6dc734b049ecc0ade547ed01a P 6b5db6fd4e9449668a3e23baa7966788 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69c1334142ef4ff2af2cff93bfaccce7" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 33097 } } peers { permanent_uuid: "d779c413657b43cd8237b6dfd61ac2e9" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 35435 } } peers { permanent_uuid: "6b5db6fd4e9449668a3e23baa7966788" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 39849 } }
I20260621 07:47:34.430188 5577 heartbeater.cc:503] Master 127.0.192.254:46129 was elected leader, sending a full tablet report...
I20260621 07:47:34.430413 5858 ts_tablet_manager.cc:1428] T b97fc3d6dc734b049ecc0ade547ed01a P 6b5db6fd4e9449668a3e23baa7966788: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260621 07:47:34.466938 5709 tablet.cc:2369] T b97fc3d6dc734b049ecc0ade547ed01a P d779c413657b43cd8237b6dfd61ac2e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 07:47:34.556730 5578 tablet.cc:2369] T b97fc3d6dc734b049ecc0ade547ed01a P 6b5db6fd4e9449668a3e23baa7966788: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:47:34.621814 5862 raft_consensus.cc:482] T b97fc3d6dc734b049ecc0ade547ed01a P d779c413657b43cd8237b6dfd61ac2e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:47:34.622007 5862 raft_consensus.cc:504] T b97fc3d6dc734b049ecc0ade547ed01a P d779c413657b43cd8237b6dfd61ac2e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69c1334142ef4ff2af2cff93bfaccce7" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 33097 } } peers { permanent_uuid: "d779c413657b43cd8237b6dfd61ac2e9" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 35435 } } peers { permanent_uuid: "6b5db6fd4e9449668a3e23baa7966788" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 39849 } }
I20260621 07:47:34.623070 5862 leader_election.cc:288] T b97fc3d6dc734b049ecc0ade547ed01a P d779c413657b43cd8237b6dfd61ac2e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69c1334142ef4ff2af2cff93bfaccce7 (127.0.192.195:33097), 6b5db6fd4e9449668a3e23baa7966788 (127.0.192.193:39849)
I20260621 07:47:34.624987 5710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:34.623348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.195:33097 (local address 127.0.192.194:54941)
0621 07:47:34.623490 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:34.623507 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:34.623609 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:34.623951 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:34.623954 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:34.623969 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:34.624430 (+ 461us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:47:34.624432 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:34.624448 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:47:34.624471 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:34.624787 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:34.624789 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:34.624791 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 07:47:34.624859 (+ 68us) client_negotiation.cc:240] Negotiation successful
0621 07:47:34.624897 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260621 07:47:34.625229 5851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:34.623415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:54941 (local address 127.0.192.195:33097)
0621 07:47:34.623551 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:34.623558 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:34.623575 (+ 17us) server_negotiation.cc:407] Connection header received
0621 07:47:34.623745 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:34.623751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:34.623799 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:34.623887 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:34.624617 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:34.624620 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:34.624623 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:47:34.624639 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:34.624673 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:47:34.624686 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:34.624776 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:34.625066 (+ 290us) server_negotiation.cc:299] Negotiation successful
0621 07:47:34.625110 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 07:47:34.625243 5855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:34.623763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:37957 (local address 127.0.192.193:39849)
0621 07:47:34.623916 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:34.623921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:34.624074 (+ 153us) server_negotiation.cc:407] Connection header received
0621 07:47:34.624181 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:34.624184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:34.624217 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:34.624300 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:34.624883 (+ 583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:34.624886 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:34.624889 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:47:34.624900 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:34.624916 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:47:34.624924 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:34.625018 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:34.625110 (+ 92us) server_negotiation.cc:299] Negotiation successful
0621 07:47:34.625144 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260621 07:47:34.625386 5840 tablet.cc:2369] T b97fc3d6dc734b049ecc0ade547ed01a P 69c1334142ef4ff2af2cff93bfaccce7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:47:34.625568 5865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:34.623629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:39849 (local address 127.0.192.194:37957)
0621 07:47:34.623976 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:34.624000 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:34.624093 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:34.624331 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:34.624334 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:34.624351 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:34.624751 (+ 400us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:47:34.624753 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:34.624762 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:47:34.624787 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:34.625005 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:34.625008 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:34.625010 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 07:47:34.625373 (+ 363us) client_negotiation.cc:240] Negotiation successful
0621 07:47:34.625411 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":127,"threads_started":1}
I20260621 07:47:34.625772 5793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b97fc3d6dc734b049ecc0ade547ed01a" candidate_uuid: "d779c413657b43cd8237b6dfd61ac2e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69c1334142ef4ff2af2cff93bfaccce7" is_pre_election: true
I20260621 07:47:34.626042 5793 raft_consensus.cc:2454] T b97fc3d6dc734b049ecc0ade547ed01a P 69c1334142ef4ff2af2cff93bfaccce7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d779c413657b43cd8237b6dfd61ac2e9 in term 0.
I20260621 07:47:34.626394 5531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b97fc3d6dc734b049ecc0ade547ed01a" candidate_uuid: "d779c413657b43cd8237b6dfd61ac2e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b5db6fd4e9449668a3e23baa7966788" is_pre_election: true
I20260621 07:47:34.626549 5595 leader_election.cc:302] T b97fc3d6dc734b049ecc0ade547ed01a P d779c413657b43cd8237b6dfd61ac2e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69c1334142ef4ff2af2cff93bfaccce7, d779c413657b43cd8237b6dfd61ac2e9; no voters:
I20260621 07:47:34.626645 5531 raft_consensus.cc:2454] T b97fc3d6dc734b049ecc0ade547ed01a P 6b5db6fd4e9449668a3e23baa7966788 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d779c413657b43cd8237b6dfd61ac2e9 in term 0.
I20260621 07:47:34.626844 5862 raft_consensus.cc:2781] T b97fc3d6dc734b049ecc0ade547ed01a P d779c413657b43cd8237b6dfd61ac2e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:47:34.626924 5862 raft_consensus.cc:482] T b97fc3d6dc734b049ecc0ade547ed01a P d779c413657b43cd8237b6dfd61ac2e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:47:34.626981 5862 raft_consensus.cc:3037] T b97fc3d6dc734b049ecc0ade547ed01a P d779c413657b43cd8237b6dfd61ac2e9 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:34.628026 5862 raft_consensus.cc:504] T b97fc3d6dc734b049ecc0ade547ed01a P d779c413657b43cd8237b6dfd61ac2e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69c1334142ef4ff2af2cff93bfaccce7" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 33097 } } peers { permanent_uuid: "d779c413657b43cd8237b6dfd61ac2e9" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 35435 } } peers { permanent_uuid: "6b5db6fd4e9449668a3e23baa7966788" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 39849 } }
I20260621 07:47:34.628309 5862 leader_election.cc:288] T b97fc3d6dc734b049ecc0ade547ed01a P d779c413657b43cd8237b6dfd61ac2e9 [CANDIDATE]: Term 1 election: Requested vote from peers 69c1334142ef4ff2af2cff93bfaccce7 (127.0.192.195:33097), 6b5db6fd4e9449668a3e23baa7966788 (127.0.192.193:39849)
I20260621 07:47:34.628624 5793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b97fc3d6dc734b049ecc0ade547ed01a" candidate_uuid: "d779c413657b43cd8237b6dfd61ac2e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69c1334142ef4ff2af2cff93bfaccce7"
I20260621 07:47:34.628640 5531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b97fc3d6dc734b049ecc0ade547ed01a" candidate_uuid: "d779c413657b43cd8237b6dfd61ac2e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b5db6fd4e9449668a3e23baa7966788"
I20260621 07:47:34.628736 5793 raft_consensus.cc:3037] T b97fc3d6dc734b049ecc0ade547ed01a P 69c1334142ef4ff2af2cff93bfaccce7 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:34.628736 5531 raft_consensus.cc:3037] T b97fc3d6dc734b049ecc0ade547ed01a P 6b5db6fd4e9449668a3e23baa7966788 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:34.629896 5793 raft_consensus.cc:2454] T b97fc3d6dc734b049ecc0ade547ed01a P 69c1334142ef4ff2af2cff93bfaccce7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d779c413657b43cd8237b6dfd61ac2e9 in term 1.
I20260621 07:47:34.629896 5531 raft_consensus.cc:2454] T b97fc3d6dc734b049ecc0ade547ed01a P 6b5db6fd4e9449668a3e23baa7966788 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d779c413657b43cd8237b6dfd61ac2e9 in term 1.
I20260621 07:47:34.630119 5595 leader_election.cc:302] T b97fc3d6dc734b049ecc0ade547ed01a P d779c413657b43cd8237b6dfd61ac2e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69c1334142ef4ff2af2cff93bfaccce7, d779c413657b43cd8237b6dfd61ac2e9; no voters:
I20260621 07:47:34.630282 5862 raft_consensus.cc:2781] T b97fc3d6dc734b049ecc0ade547ed01a P d779c413657b43cd8237b6dfd61ac2e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:47:34.630492 5862 raft_consensus.cc:686] T b97fc3d6dc734b049ecc0ade547ed01a P d779c413657b43cd8237b6dfd61ac2e9 [term 1 LEADER]: Becoming Leader. State: Replica: d779c413657b43cd8237b6dfd61ac2e9, State: Running, Role: LEADER
I20260621 07:47:34.630815 5862 consensus_queue.cc:237] T b97fc3d6dc734b049ecc0ade547ed01a P d779c413657b43cd8237b6dfd61ac2e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69c1334142ef4ff2af2cff93bfaccce7" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 33097 } } peers { permanent_uuid: "d779c413657b43cd8237b6dfd61ac2e9" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 35435 } } peers { permanent_uuid: "6b5db6fd4e9449668a3e23baa7966788" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 39849 } }
I20260621 07:47:34.633824 5388 catalog_manager.cc:5495] T b97fc3d6dc734b049ecc0ade547ed01a P d779c413657b43cd8237b6dfd61ac2e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to d779c413657b43cd8237b6dfd61ac2e9 (127.0.192.194). New cstate: current_term: 1 leader_uuid: "d779c413657b43cd8237b6dfd61ac2e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69c1334142ef4ff2af2cff93bfaccce7" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 33097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d779c413657b43cd8237b6dfd61ac2e9" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 35435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b5db6fd4e9449668a3e23baa7966788" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 39849 } health_report { overall_health: UNKNOWN } } }
I20260621 07:47:34.695561 5853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:34.694308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56350 (local address 127.0.192.194:35435)
0621 07:47:34.694433 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:34.694437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:34.694450 (+ 13us) server_negotiation.cc:407] Connection header received
0621 07:47:34.694517 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:34.694519 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:34.694572 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:34.694642 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:34.695325 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:34.695327 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:34.695329 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:47:34.695338 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:34.695350 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:47:34.695356 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:34.695413 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:34.695446 (+ 33us) server_negotiation.cc:299] Negotiation successful
0621 07:47:34.695473 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 07:47:34.700712 5793 raft_consensus.cc:1261] T b97fc3d6dc734b049ecc0ade547ed01a P 69c1334142ef4ff2af2cff93bfaccce7 [term 1 FOLLOWER]: Refusing update from remote peer d779c413657b43cd8237b6dfd61ac2e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:47:34.700712 5531 raft_consensus.cc:1261] T b97fc3d6dc734b049ecc0ade547ed01a P 6b5db6fd4e9449668a3e23baa7966788 [term 1 FOLLOWER]: Refusing update from remote peer d779c413657b43cd8237b6dfd61ac2e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:47:34.701365 5866 consensus_queue.cc:1035] T b97fc3d6dc734b049ecc0ade547ed01a P d779c413657b43cd8237b6dfd61ac2e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69c1334142ef4ff2af2cff93bfaccce7" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 33097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:47:34.701555 5862 consensus_queue.cc:1035] T b97fc3d6dc734b049ecc0ade547ed01a P d779c413657b43cd8237b6dfd61ac2e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b5db6fd4e9449668a3e23baa7966788" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:47:34.707973 5870 mvcc.cc:205] Tried to move back new op lower bound from 7299186912047763456 to 7299186911769784320. Current Snapshot: MvccSnapshot[applied={T|T < 7299186912047763456}]
I20260621 07:47:34.715377 5389 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:47:34.715523 5389 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:47:34.717717 5389 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 52706a9c0cc14e40b375840876d6adce: Sending DeleteTablet for 3 replicas of tablet b97fc3d6dc734b049ecc0ade547ed01a
I20260621 07:47:34.718386 5773 tablet_service.cc:1514] Processing DeleteTablet for tablet b97fc3d6dc734b049ecc0ade547ed01a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:47:34 UTC) from {username='slave'} at 127.0.0.1:34224
I20260621 07:47:34.718631 5642 tablet_service.cc:1514] Processing DeleteTablet for tablet b97fc3d6dc734b049ecc0ade547ed01a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:47:34 UTC) from {username='slave'} at 127.0.0.1:56342
I20260621 07:47:34.718797 5878 tablet_replica.cc:330] T b97fc3d6dc734b049ecc0ade547ed01a P 69c1334142ef4ff2af2cff93bfaccce7: stopping tablet replica
I20260621 07:47:34.719055 5878 raft_consensus.cc:2229] T b97fc3d6dc734b049ecc0ade547ed01a P 69c1334142ef4ff2af2cff93bfaccce7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:47:34.719033 5511 tablet_service.cc:1514] Processing DeleteTablet for tablet b97fc3d6dc734b049ecc0ade547ed01a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:47:34 UTC) from {username='slave'} at 127.0.0.1:58792
I20260621 07:47:34.719085 5879 tablet_replica.cc:330] T b97fc3d6dc734b049ecc0ade547ed01a P d779c413657b43cd8237b6dfd61ac2e9: stopping tablet replica
I20260621 07:47:34.719364 5878 raft_consensus.cc:2258] T b97fc3d6dc734b049ecc0ade547ed01a P 69c1334142ef4ff2af2cff93bfaccce7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:47:34.719374 5879 raft_consensus.cc:2229] T b97fc3d6dc734b049ecc0ade547ed01a P d779c413657b43cd8237b6dfd61ac2e9 [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:47:34.719678 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 5449
I20260621 07:47:34.719820 5879 raft_consensus.cc:2258] T b97fc3d6dc734b049ecc0ade547ed01a P d779c413657b43cd8237b6dfd61ac2e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:47:34.720964 5878 ts_tablet_manager.cc:1905] T b97fc3d6dc734b049ecc0ade547ed01a P 69c1334142ef4ff2af2cff93bfaccce7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:47:34.721411 5879 ts_tablet_manager.cc:1905] T b97fc3d6dc734b049ecc0ade547ed01a P d779c413657b43cd8237b6dfd61ac2e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:47:34.723856 5879 ts_tablet_manager.cc:1918] T b97fc3d6dc734b049ecc0ade547ed01a P d779c413657b43cd8237b6dfd61ac2e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:47:34.723956 5878 ts_tablet_manager.cc:1918] T b97fc3d6dc734b049ecc0ade547ed01a P 69c1334142ef4ff2af2cff93bfaccce7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:47:34.723990 5879 log.cc:1196] T b97fc3d6dc734b049ecc0ade547ed01a P d779c413657b43cd8237b6dfd61ac2e9: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-1/wal/wals/b97fc3d6dc734b049ecc0ade547ed01a
I20260621 07:47:34.724123 5878 log.cc:1196] T b97fc3d6dc734b049ecc0ade547ed01a P 69c1334142ef4ff2af2cff93bfaccce7: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782028028522580-771-0/minicluster-data/ts-2/wal/wals/b97fc3d6dc734b049ecc0ade547ed01a
I20260621 07:47:34.724279 5879 ts_tablet_manager.cc:1939] T b97fc3d6dc734b049ecc0ade547ed01a P d779c413657b43cd8237b6dfd61ac2e9: Deleting consensus metadata
I20260621 07:47:34.724543 5878 ts_tablet_manager.cc:1939] T b97fc3d6dc734b049ecc0ade547ed01a P 69c1334142ef4ff2af2cff93bfaccce7: Deleting consensus metadata
I20260621 07:47:34.725366 5373 catalog_manager.cc:4841] TS d779c413657b43cd8237b6dfd61ac2e9 (127.0.192.194:35435): tablet b97fc3d6dc734b049ecc0ade547ed01a (table test-table [id=e4125d58ae1f443da639ae79dd55109e]) successfully deleted
I20260621 07:47:34.725796 5373 catalog_manager.cc:4841] TS 69c1334142ef4ff2af2cff93bfaccce7 (127.0.192.195:33097): tablet b97fc3d6dc734b049ecc0ade547ed01a (table test-table [id=e4125d58ae1f443da639ae79dd55109e]) successfully deleted
W20260621 07:47:34.728335 5373 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.192.193:39849 (error 108)
I20260621 07:47:34.728502 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 5580
I20260621 07:47:34.729732 5854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:34.729482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:39849 (local address 127.0.0.1:58808)
0621 07:47:34.729600 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:34.729668 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.193:39849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260621 07:47:34.729935 5373 catalog_manager.cc:4584] TS 6b5db6fd4e9449668a3e23baa7966788 (127.0.192.193:39849): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b97fc3d6dc734b049ecc0ade547ed01a: Network error: Client connection negotiation failed: client connection to 127.0.192.193:39849: connect: Connection refused (error 111)
I20260621 07:47:34.733801 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 5711
I20260621 07:47:34.738197 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 5358
2026-06-21T07:47:34Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1063 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260621 07:47:34.757411 771 mini_oidc.cc:152] Starting JWKS server
I20260621 07:47:34.758872 771 webserver.cc:460] Webserver started at https://127.0.0.1:41433/ using document root <none> and password file <none>
I20260621 07:47:34.758982 771 mini_oidc.cc:189] Starting OIDC Discovery server
I20260621 07:47:34.759163 771 webserver.cc:460] Webserver started at http://127.0.0.1:39723/ using document root <none> and password file <none>
2026-06-21T07:47:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:47:34Z Disabled control of system clock
I20260621 07:47:34.766395 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:43535
--webserver_interface=127.0.192.254
--webserver_port=0
--builtin_ntp_servers=127.0.192.212:43379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:43535
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41433/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782028028522580-771-0/testchainca.cert with env {}
W20260621 07:47:34.862990 5887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:34.863224 5887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:34.863286 5887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:34.866887 5887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:47:34.866958 5887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:34.866986 5887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:34.867022 5887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:47:34.867053 5887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260621 07:47:34.867097 5887 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260621 07:47:34.867129 5887 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41433/jwks.json
I20260621 07:47:34.875545 5887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:43379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:43535
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:43535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41433/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782028028522580-771-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:34.876799 5887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:34.878068 5887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:34.888190 5895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:34.888185 5893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:34.888756 5892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:34.889232 5887 server_base.cc:1034] running on GCE node
I20260621 07:47:34.889835 5887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:34.891067 5887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:34.892233 5887 hybrid_clock.cc:648] HybridClock initialized: now 1782028054892200 us; error 47 us; skew 500 ppm
I20260621 07:47:34.894255 5887 webserver.cc:460] Webserver started at http://127.0.192.254:37181/ using document root <none> and password file <none>
I20260621 07:47:34.894778 5887 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:34.894883 5887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:34.895104 5887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:34.897823 5887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "a950619d34814be1810c446863a19a29"
format_stamp: "Formatted at 2026-06-21 07:47:34 on dist-test-slave-lzx3"
server_key: "165cc5a2d905416c1c27baa78b5ddb02"
server_key_iv: "53779bcebf1567508c522ad22a4afe1e"
server_key_version: "encryptionkey@0"
I20260621 07:47:34.898494 5887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "a950619d34814be1810c446863a19a29"
format_stamp: "Formatted at 2026-06-21 07:47:34 on dist-test-slave-lzx3"
server_key: "165cc5a2d905416c1c27baa78b5ddb02"
server_key_iv: "53779bcebf1567508c522ad22a4afe1e"
server_key_version: "encryptionkey@0"
I20260621 07:47:34.903432 5887 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 07:47:34.906473 5901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:34.907944 5887 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 07:47:34.908046 5887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "a950619d34814be1810c446863a19a29"
format_stamp: "Formatted at 2026-06-21 07:47:34 on dist-test-slave-lzx3"
server_key: "165cc5a2d905416c1c27baa78b5ddb02"
server_key_iv: "53779bcebf1567508c522ad22a4afe1e"
server_key_version: "encryptionkey@0"
I20260621 07:47:34.908149 5887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:34.917286 5887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:34.917840 5887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:34.917989 5887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:34.923696 5887 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:43535
I20260621 07:47:34.923702 5953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:43535 every 8 connection(s)
I20260621 07:47:34.924455 5887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:47:34.927376 5954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:34.930867 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 5887
I20260621 07:47:34.930953 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:47:34.931206 771 external_mini_cluster.cc:1428] Setting key 3c76ef88f32f6b46360d908da177f128
I20260621 07:47:34.932339 5954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a950619d34814be1810c446863a19a29: Bootstrap starting.
I20260621 07:47:34.934315 5954 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a950619d34814be1810c446863a19a29: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:34.935001 5954 log.cc:824] T 00000000000000000000000000000000 P a950619d34814be1810c446863a19a29: Log is configured to *not* fsync() on all Append() calls
I20260621 07:47:34.936709 5954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a950619d34814be1810c446863a19a29: No bootstrap required, opened a new log
I20260621 07:47:34.939410 5957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:34.932364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49760 (local address 127.0.192.254:43535)
0621 07:47:34.932920 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:34.932934 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:34.932964 (+ 30us) server_negotiation.cc:407] Connection header received
0621 07:47:34.933484 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:34.933497 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:34.933810 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:34.934123 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:34.934559 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:34.935645 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:34.936475 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:34.937050 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:34.937211 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:34.937976 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:34.937981 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:34.938001 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:47:34.938337 (+ 336us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:34.938380 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:47:34.938397 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:34.938486 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:34.938580 (+ 94us) server_negotiation.cc:299] Negotiation successful
0621 07:47:34.938804 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":125,"threads_started":1}
I20260621 07:47:34.939893 5954 raft_consensus.cc:348] T 00000000000000000000000000000000 P a950619d34814be1810c446863a19a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a950619d34814be1810c446863a19a29" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 43535 } }
I20260621 07:47:34.940096 5954 raft_consensus.cc:374] T 00000000000000000000000000000000 P a950619d34814be1810c446863a19a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:34.940166 5954 raft_consensus.cc:729] T 00000000000000000000000000000000 P a950619d34814be1810c446863a19a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a950619d34814be1810c446863a19a29, State: Initialized, Role: FOLLOWER
I20260621 07:47:34.940650 5954 consensus_queue.cc:260] T 00000000000000000000000000000000 P a950619d34814be1810c446863a19a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a950619d34814be1810c446863a19a29" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 43535 } }
I20260621 07:47:34.940790 5954 raft_consensus.cc:388] T 00000000000000000000000000000000 P a950619d34814be1810c446863a19a29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:47:34.940860 5954 raft_consensus.cc:482] T 00000000000000000000000000000000 P a950619d34814be1810c446863a19a29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:47:34.940958 5954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a950619d34814be1810c446863a19a29 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:34.941844 771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:47:34.941916 5954 raft_consensus.cc:504] T 00000000000000000000000000000000 P a950619d34814be1810c446863a19a29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a950619d34814be1810c446863a19a29" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 43535 } }
I20260621 07:47:34.942199 5954 leader_election.cc:302] T 00000000000000000000000000000000 P a950619d34814be1810c446863a19a29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a950619d34814be1810c446863a19a29; no voters:
I20260621 07:47:34.942488 5954 leader_election.cc:288] T 00000000000000000000000000000000 P a950619d34814be1810c446863a19a29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:47:34.942559 5959 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a950619d34814be1810c446863a19a29 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:47:34.942759 5959 raft_consensus.cc:686] T 00000000000000000000000000000000 P a950619d34814be1810c446863a19a29 [term 1 LEADER]: Becoming Leader. State: Replica: a950619d34814be1810c446863a19a29, State: Running, Role: LEADER
I20260621 07:47:34.943027 5959 consensus_queue.cc:237] T 00000000000000000000000000000000 P a950619d34814be1810c446863a19a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a950619d34814be1810c446863a19a29" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 43535 } }
I20260621 07:47:34.943632 5954 sys_catalog.cc:564] T 00000000000000000000000000000000 P a950619d34814be1810c446863a19a29 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:47:34.944682 5961 sys_catalog.cc:455] T 00000000000000000000000000000000 P a950619d34814be1810c446863a19a29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a950619d34814be1810c446863a19a29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a950619d34814be1810c446863a19a29" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 43535 } } }
I20260621 07:47:34.944689 5960 sys_catalog.cc:455] T 00000000000000000000000000000000 P a950619d34814be1810c446863a19a29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a950619d34814be1810c446863a19a29. Latest consensus state: current_term: 1 leader_uuid: "a950619d34814be1810c446863a19a29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a950619d34814be1810c446863a19a29" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 43535 } } }
I20260621 07:47:34.944813 5961 sys_catalog.cc:458] T 00000000000000000000000000000000 P a950619d34814be1810c446863a19a29 [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:34.944836 5960 sys_catalog.cc:458] T 00000000000000000000000000000000 P a950619d34814be1810c446863a19a29 [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:34.945899 5968 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:47:34.948280 5968 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:47:34.953442 5968 catalog_manager.cc:1347] Generated new cluster ID: 20fa5124cc1b48dab70b7a9934b7cc26
I20260621 07:47:34.953521 5968 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:47:34.962062 5968 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:47:34.962790 5968 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:47:34.975517 5968 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a950619d34814be1810c446863a19a29: Generated new TSK 0
I20260621 07:47:34.976181 5968 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260621 07:47:35.004976 5984 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260621 07:47:35.005069 5984 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260621 07:47:35.005478 5957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:34.984031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49768 (local address 127.0.192.254:43535)
0621 07:47:34.984152 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:34.984156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:34.984406 (+ 250us) server_negotiation.cc:407] Connection header received
0621 07:47:34.984653 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:34.984656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:34.984697 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:34.984771 (+ 74us) server_negotiation.cc:226] Negotiated authn=JWT
0621 07:47:34.985037 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:34.985556 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:34.986799 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:34.987013 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:34.987086 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:34.987446 (+ 360us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0621 07:47:35.005117 (+ 17671us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41433/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0621 07:47:35.005344 (+ 227us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49768: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41433/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":49}
W20260621 07:47:35.005559 5957 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49768: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41433/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260621 07:47:35.006575 5983 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:34.983968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:43535 (local address 127.0.0.1:49768)
0621 07:47:34.984295 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:34.984337 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:34.984471 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:34.984811 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:34.984815 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:34.984824 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0621 07:47:34.984939 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:34.984947 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:34.985661 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:34.985665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:34.986685 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:34.986696 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:34.987120 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:34.987124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:34.987221 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:34.987275 (+ 54us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0621 07:47:35.005303 (+ 18028us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41433/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0621 07:47:35.005531 (+ 228us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.192.254:43535: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41433/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":98,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.192.254:43535: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41433/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260621 07:47:35.008317 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 5887
2026-06-21T07:47:35Z chronyd exiting
I20260621 07:47:35.044732 771 test_util.cc:175] -----------------------------------------------
I20260621 07:47:35.044788 771 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782028028522580-771-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (288 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260621 07:47:35.045676 771 mini_oidc.cc:152] Starting JWKS server
I20260621 07:47:35.046471 771 webserver.cc:460] Webserver started at https://127.0.0.1:33141/ using document root <none> and password file <none>
I20260621 07:47:35.046581 771 mini_oidc.cc:189] Starting OIDC Discovery server
I20260621 07:47:35.046866 771 webserver.cc:460] Webserver started at http://127.0.0.1:44869/ using document root <none> and password file <none>
2026-06-21T07:47:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:47:35Z Disabled control of system clock
I20260621 07:47:35.054646 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:45993
--webserver_interface=127.0.192.254
--webserver_port=0
--builtin_ntp_servers=127.0.192.212:46857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:45993
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33141/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782028028522580-771-0/testchainca.cert with env {}
W20260621 07:47:35.150803 5990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:35.151002 5990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:35.151039 5990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:35.154460 5990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:47:35.154506 5990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:35.154524 5990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:35.154543 5990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:47:35.154582 5990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260621 07:47:35.154603 5990 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260621 07:47:35.154632 5990 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33141/jwks.json
I20260621 07:47:35.159276 5990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:46857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:45993
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:45993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33141/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782028028522580-771-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:35.160020 5990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:35.160758 5990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:35.168102 5995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:35.168191 5990 server_base.cc:1034] running on GCE node
W20260621 07:47:35.168102 5996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:35.168154 5998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:35.168833 5990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:35.169610 5990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:35.170753 5990 hybrid_clock.cc:648] HybridClock initialized: now 1782028055170736 us; error 27 us; skew 500 ppm
I20260621 07:47:35.172328 5990 webserver.cc:460] Webserver started at http://127.0.192.254:37613/ using document root <none> and password file <none>
I20260621 07:47:35.172869 5990 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:35.172926 5990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:35.173126 5990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:35.174810 5990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "67a6792a84f1406b84749e450f4a021b"
format_stamp: "Formatted at 2026-06-21 07:47:35 on dist-test-slave-lzx3"
server_key: "b8ae4f6b7b4e9752bd745a7bab68bdc7"
server_key_iv: "b0a897b65fc16a8f06f8382e5ce9ccb5"
server_key_version: "encryptionkey@0"
I20260621 07:47:35.175340 5990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "67a6792a84f1406b84749e450f4a021b"
format_stamp: "Formatted at 2026-06-21 07:47:35 on dist-test-slave-lzx3"
server_key: "b8ae4f6b7b4e9752bd745a7bab68bdc7"
server_key_iv: "b0a897b65fc16a8f06f8382e5ce9ccb5"
server_key_version: "encryptionkey@0"
I20260621 07:47:35.178531 5990 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 07:47:35.180478 6004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:35.181521 5990 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:35.181629 5990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "67a6792a84f1406b84749e450f4a021b"
format_stamp: "Formatted at 2026-06-21 07:47:35 on dist-test-slave-lzx3"
server_key: "b8ae4f6b7b4e9752bd745a7bab68bdc7"
server_key_iv: "b0a897b65fc16a8f06f8382e5ce9ccb5"
server_key_version: "encryptionkey@0"
I20260621 07:47:35.181718 5990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:35.192894 5990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:35.193361 5990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:35.193524 5990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:35.199132 5990 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:45993
I20260621 07:47:35.199141 6056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:45993 every 8 connection(s)
I20260621 07:47:35.199873 5990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:47:35.202569 6057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:35.207037 6057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67a6792a84f1406b84749e450f4a021b: Bootstrap starting.
I20260621 07:47:35.209143 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 5990
I20260621 07:47:35.209247 6057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67a6792a84f1406b84749e450f4a021b: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:35.209275 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:47:35.209565 771 external_mini_cluster.cc:1428] Setting key 928465415164bd78975e7051814297ed
I20260621 07:47:35.210004 6057 log.cc:824] T 00000000000000000000000000000000 P 67a6792a84f1406b84749e450f4a021b: Log is configured to *not* fsync() on all Append() calls
I20260621 07:47:35.211625 6057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67a6792a84f1406b84749e450f4a021b: No bootstrap required, opened a new log
I20260621 07:47:35.215291 6057 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67a6792a84f1406b84749e450f4a021b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a6792a84f1406b84749e450f4a021b" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45993 } }
I20260621 07:47:35.215554 6057 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67a6792a84f1406b84749e450f4a021b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:35.215624 6057 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67a6792a84f1406b84749e450f4a021b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67a6792a84f1406b84749e450f4a021b, State: Initialized, Role: FOLLOWER
I20260621 07:47:35.216001 6057 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67a6792a84f1406b84749e450f4a021b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a6792a84f1406b84749e450f4a021b" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45993 } }
I20260621 07:47:35.216101 6057 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67a6792a84f1406b84749e450f4a021b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:47:35.216171 6057 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67a6792a84f1406b84749e450f4a021b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:47:35.216297 6057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67a6792a84f1406b84749e450f4a021b [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:35.217310 6057 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67a6792a84f1406b84749e450f4a021b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a6792a84f1406b84749e450f4a021b" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45993 } }
I20260621 07:47:35.217550 6061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:35.210425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55270 (local address 127.0.192.254:45993)
0621 07:47:35.210870 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:35.210884 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:35.210913 (+ 29us) server_negotiation.cc:407] Connection header received
0621 07:47:35.211402 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:35.211415 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:35.211722 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:35.211965 (+ 243us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:35.212231 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:35.213124 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:35.214270 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:35.214684 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:35.214794 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:35.215568 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:35.215576 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:35.215595 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:47:35.216041 (+ 446us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:35.216105 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:47:35.216136 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:35.216239 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:35.216404 (+ 165us) server_negotiation.cc:299] Negotiation successful
0621 07:47:35.216675 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":93,"threads_started":1}
I20260621 07:47:35.217654 6057 leader_election.cc:302] T 00000000000000000000000000000000 P 67a6792a84f1406b84749e450f4a021b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67a6792a84f1406b84749e450f4a021b; no voters:
I20260621 07:47:35.217919 6057 leader_election.cc:288] T 00000000000000000000000000000000 P 67a6792a84f1406b84749e450f4a021b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:47:35.218019 6062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67a6792a84f1406b84749e450f4a021b [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:47:35.218225 6062 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67a6792a84f1406b84749e450f4a021b [term 1 LEADER]: Becoming Leader. State: Replica: 67a6792a84f1406b84749e450f4a021b, State: Running, Role: LEADER
I20260621 07:47:35.218488 6062 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67a6792a84f1406b84749e450f4a021b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a6792a84f1406b84749e450f4a021b" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45993 } }
I20260621 07:47:35.219020 6057 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67a6792a84f1406b84749e450f4a021b [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:47:35.219435 6063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67a6792a84f1406b84749e450f4a021b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67a6792a84f1406b84749e450f4a021b. Latest consensus state: current_term: 1 leader_uuid: "67a6792a84f1406b84749e450f4a021b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a6792a84f1406b84749e450f4a021b" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45993 } } }
I20260621 07:47:35.220112 6063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67a6792a84f1406b84749e450f4a021b [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:35.219996 6064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67a6792a84f1406b84749e450f4a021b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67a6792a84f1406b84749e450f4a021b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a6792a84f1406b84749e450f4a021b" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45993 } } }
I20260621 07:47:35.220357 771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:47:35.220398 6064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67a6792a84f1406b84749e450f4a021b [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:35.221230 6071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:47:35.223460 6071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:47:35.227803 6071 catalog_manager.cc:1347] Generated new cluster ID: 2719956e6c0441678d1be30cf9165e1d
I20260621 07:47:35.227859 6071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:47:35.233443 6071 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:47:35.234237 6071 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:47:35.240446 6071 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67a6792a84f1406b84749e450f4a021b: Generated new TSK 0
I20260621 07:47:35.240945 6071 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260621 07:47:35.262007 6087 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260621 07:47:35.262063 6087 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260621 07:47:35.262552 6061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:35.248850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55284 (local address 127.0.192.254:45993)
0621 07:47:35.248957 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:35.248961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:35.249153 (+ 192us) server_negotiation.cc:407] Connection header received
0621 07:47:35.249303 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:35.249305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:35.249349 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:35.249448 (+ 99us) server_negotiation.cc:226] Negotiated authn=JWT
0621 07:47:35.249651 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:35.250082 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:35.250862 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:35.251068 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:35.251160 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:35.251437 (+ 277us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0621 07:47:35.262258 (+ 10821us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33141/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0621 07:47:35.262448 (+ 190us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55284: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33141/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":45}
W20260621 07:47:35.262602 6061 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55284: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33141/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260621 07:47:35.262652 6086 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:35.248777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45993 (local address 127.0.0.1:55284)
0621 07:47:35.249075 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:35.249085 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:35.249193 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:35.249449 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:35.249452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:35.249459 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0621 07:47:35.249558 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:35.249563 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:35.250199 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:35.250203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:35.250735 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:35.250758 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:35.251173 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:35.251176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:35.251257 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:35.251277 (+ 20us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0621 07:47:35.262384 (+ 11107us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33141/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0621 07:47:35.262504 (+ 120us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.192.254:45993: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33141/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":97,"threads_started":1}
I20260621 07:47:35.263242 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 5990
2026-06-21T07:47:35Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (255 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260621 07:47:35.301752 771 mini_oidc.cc:152] Starting JWKS server
I20260621 07:47:35.302515 771 webserver.cc:460] Webserver started at https://127.0.0.1:39771/ using document root <none> and password file <none>
I20260621 07:47:35.302588 771 mini_oidc.cc:189] Starting OIDC Discovery server
I20260621 07:47:35.302784 771 webserver.cc:460] Webserver started at http://127.0.0.1:38261/ using document root <none> and password file <none>
2026-06-21T07:47:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:47:35Z Disabled control of system clock
I20260621 07:47:35.310864 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:35789
--webserver_interface=127.0.192.254
--webserver_port=0
--builtin_ntp_servers=127.0.192.212:36755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:35789
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39771/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260621 07:47:35.409271 6093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:35.409456 6093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:35.409488 6093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:35.412698 6093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:47:35.412734 6093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:35.412746 6093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:35.412760 6093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:47:35.412773 6093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260621 07:47:35.412791 6093 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260621 07:47:35.412808 6093 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39771/jwks.json
I20260621 07:47:35.416888 6093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:36755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:35789
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:35789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39771/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:35.417644 6093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:35.418650 6093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:35.425431 6101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:35.425459 6099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:35.425431 6098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:35.425711 6093 server_base.cc:1034] running on GCE node
I20260621 07:47:35.426129 6093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:35.427029 6093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:35.428174 6093 hybrid_clock.cc:648] HybridClock initialized: now 1782028055428152 us; error 36 us; skew 500 ppm
I20260621 07:47:35.429630 6093 webserver.cc:460] Webserver started at http://127.0.192.254:43447/ using document root <none> and password file <none>
I20260621 07:47:35.430068 6093 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:35.430117 6093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:35.430313 6093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:35.431875 6093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "f2e779787b924123a42ec84cfcc5240f"
format_stamp: "Formatted at 2026-06-21 07:47:35 on dist-test-slave-lzx3"
server_key: "0667bece875a1249e5cca916221eb3fe"
server_key_iv: "d473e4c3325fd6aaddb5bbced08c243b"
server_key_version: "encryptionkey@0"
I20260621 07:47:35.432318 6093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "f2e779787b924123a42ec84cfcc5240f"
format_stamp: "Formatted at 2026-06-21 07:47:35 on dist-test-slave-lzx3"
server_key: "0667bece875a1249e5cca916221eb3fe"
server_key_iv: "d473e4c3325fd6aaddb5bbced08c243b"
server_key_version: "encryptionkey@0"
I20260621 07:47:35.435417 6093 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260621 07:47:35.437487 6107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:35.438495 6093 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 07:47:35.438616 6093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "f2e779787b924123a42ec84cfcc5240f"
format_stamp: "Formatted at 2026-06-21 07:47:35 on dist-test-slave-lzx3"
server_key: "0667bece875a1249e5cca916221eb3fe"
server_key_iv: "d473e4c3325fd6aaddb5bbced08c243b"
server_key_version: "encryptionkey@0"
I20260621 07:47:35.438707 6093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:35.449460 6093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:35.449921 6093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:35.450067 6093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:35.456178 6159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:35789 every 8 connection(s)
I20260621 07:47:35.456176 6093 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:35789
I20260621 07:47:35.457093 6093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:47:35.459668 6160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:35.463936 6160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2e779787b924123a42ec84cfcc5240f: Bootstrap starting.
I20260621 07:47:35.465051 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 6093
I20260621 07:47:35.465165 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:47:35.465423 771 external_mini_cluster.cc:1428] Setting key 2c4d94e4ad703863cfe6833c083499d4
I20260621 07:47:35.466305 6160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2e779787b924123a42ec84cfcc5240f: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:35.467033 6160 log.cc:824] T 00000000000000000000000000000000 P f2e779787b924123a42ec84cfcc5240f: Log is configured to *not* fsync() on all Append() calls
I20260621 07:47:35.468709 6160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2e779787b924123a42ec84cfcc5240f: No bootstrap required, opened a new log
I20260621 07:47:35.471495 6160 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2e779787b924123a42ec84cfcc5240f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2e779787b924123a42ec84cfcc5240f" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 35789 } }
I20260621 07:47:35.471678 6160 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2e779787b924123a42ec84cfcc5240f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:35.471750 6160 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2e779787b924123a42ec84cfcc5240f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2e779787b924123a42ec84cfcc5240f, State: Initialized, Role: FOLLOWER
I20260621 07:47:35.472115 6160 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2e779787b924123a42ec84cfcc5240f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2e779787b924123a42ec84cfcc5240f" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 35789 } }
I20260621 07:47:35.472276 6160 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2e779787b924123a42ec84cfcc5240f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:47:35.472342 6160 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2e779787b924123a42ec84cfcc5240f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:47:35.472430 6160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2e779787b924123a42ec84cfcc5240f [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:35.472986 6163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:35.466306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35408 (local address 127.0.192.254:35789)
0621 07:47:35.466881 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:35.466894 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:35.466927 (+ 33us) server_negotiation.cc:407] Connection header received
0621 07:47:35.467658 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:35.467680 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:35.467971 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:35.468259 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:35.468658 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:35.469502 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:35.470260 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:35.470641 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:35.470779 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:35.471467 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:35.471471 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:35.471480 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:47:35.471827 (+ 347us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:35.471883 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:47:35.471909 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:35.471993 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:35.472109 (+ 116us) server_negotiation.cc:299] Negotiation successful
0621 07:47:35.472326 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":118,"threads_started":1}
I20260621 07:47:35.473167 6160 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2e779787b924123a42ec84cfcc5240f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2e779787b924123a42ec84cfcc5240f" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 35789 } }
I20260621 07:47:35.473457 6160 leader_election.cc:302] T 00000000000000000000000000000000 P f2e779787b924123a42ec84cfcc5240f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2e779787b924123a42ec84cfcc5240f; no voters:
I20260621 07:47:35.473698 6160 leader_election.cc:288] T 00000000000000000000000000000000 P f2e779787b924123a42ec84cfcc5240f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:47:35.473778 6165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2e779787b924123a42ec84cfcc5240f [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:47:35.473986 6165 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2e779787b924123a42ec84cfcc5240f [term 1 LEADER]: Becoming Leader. State: Replica: f2e779787b924123a42ec84cfcc5240f, State: Running, Role: LEADER
I20260621 07:47:35.474292 6165 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2e779787b924123a42ec84cfcc5240f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2e779787b924123a42ec84cfcc5240f" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 35789 } }
I20260621 07:47:35.474728 6160 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2e779787b924123a42ec84cfcc5240f [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:47:35.475328 6166 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2e779787b924123a42ec84cfcc5240f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2e779787b924123a42ec84cfcc5240f. Latest consensus state: current_term: 1 leader_uuid: "f2e779787b924123a42ec84cfcc5240f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2e779787b924123a42ec84cfcc5240f" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 35789 } } }
I20260621 07:47:35.475416 771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:47:35.475453 6166 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2e779787b924123a42ec84cfcc5240f [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:35.475772 6170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:47:35.477232 6167 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2e779787b924123a42ec84cfcc5240f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2e779787b924123a42ec84cfcc5240f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2e779787b924123a42ec84cfcc5240f" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 35789 } } }
I20260621 07:47:35.477373 6167 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2e779787b924123a42ec84cfcc5240f [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:35.478680 6170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:47:35.483180 6170 catalog_manager.cc:1347] Generated new cluster ID: 66e023c03d6b49b5995d6f830e79f526
I20260621 07:47:35.483234 6170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:47:35.496040 6170 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:47:35.496788 6170 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:47:35.502410 6170 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2e779787b924123a42ec84cfcc5240f: Generated new TSK 0
I20260621 07:47:35.503049 6170 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:47:35.518198 6163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:35.513739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35420 (local address 127.0.192.254:35789)
0621 07:47:35.513859 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:35.513863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:35.514021 (+ 158us) server_negotiation.cc:407] Connection header received
0621 07:47:35.514189 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:35.514191 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:35.514229 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:35.514305 (+ 76us) server_negotiation.cc:226] Negotiated authn=JWT
0621 07:47:35.514560 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:35.515049 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:35.516147 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:35.516344 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:35.516432 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:35.516734 (+ 302us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0621 07:47:35.517893 (+ 1159us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39771/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0621 07:47:35.518089 (+ 196us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35420: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39771/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":44}
W20260621 07:47:35.518244 6163 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35420: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39771/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260621 07:47:35.519877 6189 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:35.513675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:35789 (local address 127.0.0.1:35420)
0621 07:47:35.513939 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:35.513952 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:35.514054 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:35.514337 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:35.514340 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:35.514348 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0621 07:47:35.514465 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:35.514471 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:35.515163 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:35.515167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:35.516046 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:35.516054 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:35.516469 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:35.516473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:35.516580 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:35.516608 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0621 07:47:35.518050 (+ 1442us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39771/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0621 07:47:35.519725 (+ 1675us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.192.254:35789: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39771/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":1443,"thread_start_us":93,"threads_started":1}
I20260621 07:47:35.520371 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 6093
E20260621 07:47:35.527848 6190 webserver.cc:533] Webserver: sslize failed:
E20260621 07:47:35.527896 6190 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-21T07:47:35Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (256 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-21T07:47:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:47:35Z Disabled control of system clock
I20260621 07:47:35.565605 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:36727
--webserver_interface=127.0.192.254
--webserver_port=0
--builtin_ntp_servers=127.0.192.212:46479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:36727
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782028028522580-771-0/testchainca.cert with env {}
W20260621 07:47:35.663672 6194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:35.663870 6194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:35.663903 6194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:35.667544 6194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:47:35.667591 6194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:35.667632 6194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:35.667680 6194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:47:35.667711 6194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260621 07:47:35.667742 6194 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260621 07:47:35.667783 6194 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260621 07:47:35.672505 6194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:46479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:36727
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:36727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782028028522580-771-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:35.673599 6194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:35.674743 6194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:35.684361 6200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:35.684372 6199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:35.684371 6202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:35.685115 6194 server_base.cc:1034] running on GCE node
I20260621 07:47:35.685534 6194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:35.686503 6194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:35.687655 6194 hybrid_clock.cc:648] HybridClock initialized: now 1782028055687636 us; error 29 us; skew 500 ppm
I20260621 07:47:35.689194 6194 webserver.cc:460] Webserver started at http://127.0.192.254:42441/ using document root <none> and password file <none>
I20260621 07:47:35.689647 6194 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:35.689699 6194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:35.689890 6194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:35.691551 6194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "2efaf5aec68e4ce5916721ff9023391a"
format_stamp: "Formatted at 2026-06-21 07:47:35 on dist-test-slave-lzx3"
server_key: "46dbc759a9633132f968fe7a8388eccc"
server_key_iv: "a9971a3a9591c671c641c3746e4568ef"
server_key_version: "encryptionkey@0"
I20260621 07:47:35.692122 6194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "2efaf5aec68e4ce5916721ff9023391a"
format_stamp: "Formatted at 2026-06-21 07:47:35 on dist-test-slave-lzx3"
server_key: "46dbc759a9633132f968fe7a8388eccc"
server_key_iv: "a9971a3a9591c671c641c3746e4568ef"
server_key_version: "encryptionkey@0"
I20260621 07:47:35.695358 6194 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 07:47:35.697502 6208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:35.698559 6194 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:35.698675 6194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "2efaf5aec68e4ce5916721ff9023391a"
format_stamp: "Formatted at 2026-06-21 07:47:35 on dist-test-slave-lzx3"
server_key: "46dbc759a9633132f968fe7a8388eccc"
server_key_iv: "a9971a3a9591c671c641c3746e4568ef"
server_key_version: "encryptionkey@0"
I20260621 07:47:35.698764 6194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:35.709537 6194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:35.710013 6194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:35.710163 6194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:35.715997 6194 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:36727
I20260621 07:47:35.716017 6260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:36727 every 8 connection(s)
I20260621 07:47:35.716797 6194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:47:35.719318 6261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:35.719983 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 6194
I20260621 07:47:35.720103 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:47:35.720347 771 external_mini_cluster.cc:1428] Setting key 6cf1ed7383491b18d342d450a9a2c6e6
I20260621 07:47:35.724263 6261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2efaf5aec68e4ce5916721ff9023391a: Bootstrap starting.
I20260621 07:47:35.726222 6261 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2efaf5aec68e4ce5916721ff9023391a: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:35.726889 6261 log.cc:824] T 00000000000000000000000000000000 P 2efaf5aec68e4ce5916721ff9023391a: Log is configured to *not* fsync() on all Append() calls
I20260621 07:47:35.728030 6264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:35.721441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47428 (local address 127.0.192.254:36727)
0621 07:47:35.722001 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:35.722023 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:35.722051 (+ 28us) server_negotiation.cc:407] Connection header received
0621 07:47:35.722602 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:35.722616 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:35.722920 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:35.723226 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:35.723569 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:35.724318 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:35.725178 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:35.725621 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:35.725785 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:35.726537 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:35.726542 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:35.726554 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:47:35.726887 (+ 333us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:35.726936 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:47:35.726954 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:35.727066 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:35.727221 (+ 155us) server_negotiation.cc:299] Negotiation successful
0621 07:47:35.727447 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":129,"threads_started":1}
I20260621 07:47:35.728538 6261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2efaf5aec68e4ce5916721ff9023391a: No bootstrap required, opened a new log
I20260621 07:47:35.730489 771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:47:35.732421 6261 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2efaf5aec68e4ce5916721ff9023391a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2efaf5aec68e4ce5916721ff9023391a" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 36727 } }
I20260621 07:47:35.732645 6261 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2efaf5aec68e4ce5916721ff9023391a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:35.732700 6261 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2efaf5aec68e4ce5916721ff9023391a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efaf5aec68e4ce5916721ff9023391a, State: Initialized, Role: FOLLOWER
I20260621 07:47:35.733039 6261 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2efaf5aec68e4ce5916721ff9023391a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2efaf5aec68e4ce5916721ff9023391a" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 36727 } }
I20260621 07:47:35.733161 6261 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2efaf5aec68e4ce5916721ff9023391a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:47:35.733220 6261 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2efaf5aec68e4ce5916721ff9023391a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:47:35.733301 6261 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2efaf5aec68e4ce5916721ff9023391a [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:35.734030 6261 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2efaf5aec68e4ce5916721ff9023391a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2efaf5aec68e4ce5916721ff9023391a" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 36727 } }
I20260621 07:47:35.734422 6261 leader_election.cc:302] T 00000000000000000000000000000000 P 2efaf5aec68e4ce5916721ff9023391a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2efaf5aec68e4ce5916721ff9023391a; no voters:
I20260621 07:47:35.734709 6261 leader_election.cc:288] T 00000000000000000000000000000000 P 2efaf5aec68e4ce5916721ff9023391a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:47:35.734841 6266 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2efaf5aec68e4ce5916721ff9023391a [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:47:35.735045 6266 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2efaf5aec68e4ce5916721ff9023391a [term 1 LEADER]: Becoming Leader. State: Replica: 2efaf5aec68e4ce5916721ff9023391a, State: Running, Role: LEADER
I20260621 07:47:35.735404 6266 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2efaf5aec68e4ce5916721ff9023391a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2efaf5aec68e4ce5916721ff9023391a" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 36727 } }
I20260621 07:47:35.735823 6261 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2efaf5aec68e4ce5916721ff9023391a [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:47:35.737061 6268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2efaf5aec68e4ce5916721ff9023391a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2efaf5aec68e4ce5916721ff9023391a. Latest consensus state: current_term: 1 leader_uuid: "2efaf5aec68e4ce5916721ff9023391a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2efaf5aec68e4ce5916721ff9023391a" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 36727 } } }
I20260621 07:47:35.737017 6267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2efaf5aec68e4ce5916721ff9023391a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2efaf5aec68e4ce5916721ff9023391a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2efaf5aec68e4ce5916721ff9023391a" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 36727 } } }
I20260621 07:47:35.737180 6268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2efaf5aec68e4ce5916721ff9023391a [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:35.737190 6267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2efaf5aec68e4ce5916721ff9023391a [sys.catalog]: This master's current role is: LEADER
W20260621 07:47:35.740100 6281 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2efaf5aec68e4ce5916721ff9023391a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:47:35.740156 6281 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:47:35.740221 6277 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:47:35.740924 6277 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:47:35.745328 6277 catalog_manager.cc:1347] Generated new cluster ID: c8ec5c6a50e243a8b26798945450fa70
I20260621 07:47:35.745378 6277 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:47:35.757650 6277 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:47:35.758543 6277 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:47:35.770383 6277 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2efaf5aec68e4ce5916721ff9023391a: Generated new TSK 0
I20260621 07:47:35.770972 6277 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:47:35.841005 6264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:35.780114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47430 (local address 127.0.192.254:36727)
0621 07:47:35.780232 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:35.780238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:35.780385 (+ 147us) server_negotiation.cc:407] Connection header received
0621 07:47:35.780540 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:35.780542 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:35.780623 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:35.780701 (+ 78us) server_negotiation.cc:226] Negotiated authn=JWT
0621 07:47:35.780934 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:35.781427 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:35.782611 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:35.782815 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:35.782885 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:35.783189 (+ 304us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0621 07:47:35.840622 (+ 57433us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0621 07:47:35.840890 (+ 268us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47430: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":51}
W20260621 07:47:35.841100 6264 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47430: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260621 07:47:35.841544 6290 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:35.780046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:36727 (local address 127.0.0.1:47430)
0621 07:47:35.780301 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:35.780314 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:35.780414 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:35.780720 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:35.780723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:35.780732 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0621 07:47:35.780840 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:35.780847 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:35.781534 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:35.781537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:35.782502 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:35.782515 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:35.782919 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:35.782923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:35.783025 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:35.783052 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0621 07:47:35.840835 (+ 57783us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0621 07:47:35.841078 (+ 243us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.192.254:36727: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":100,"threads_started":1}
I20260621 07:47:35.842119 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 6194
2026-06-21T07:47:35Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (302 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (107 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (107 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:47:36 dist-test-slave-lzx3 krb5kdc[6302](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:47:36 dist-test-slave-lzx3 krb5kdc[6302](info): set up 2 sockets
Jun 21 07:47:36 dist-test-slave-lzx3 krb5kdc[6302](info): commencing operation
krb5kdc: starting...
W20260621 07:47:37.012495 771 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/krb5kdc/test-user.keytab.
Jun 21 07:47:37 dist-test-slave-lzx3 krb5kdc[6302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:47:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:47:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.192.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
I20260621 07:47:37.113409 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:45857
--webserver_interface=127.0.192.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:36851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:45857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:37.204660 6315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:37.204870 6315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:37.204955 6315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:37.208894 6315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:47:37.208953 6315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:37.208984 6315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:37.209017 6315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:47:37.209046 6315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:47:37.213299 6315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:36851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:45857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:45857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:37.214020 6315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:37.214826 6315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:37.222035 6321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:37.222046 6323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:37.222151 6315 server_base.cc:1034] running on GCE node
W20260621 07:47:37.222046 6320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:37.222687 6315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:37.223542 6315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:37.224714 6315 hybrid_clock.cc:648] HybridClock initialized: now 1782028057224698 us; error 29 us; skew 500 ppm
Jun 21 07:47:37 dist-test-slave-lzx3 krb5kdc[6302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:37.231010 6315 init.cc:377] Logged in from keytab as kudu/127.0.192.254@KRBTEST.COM (short username kudu)
I20260621 07:47:37.231737 6315 webserver.cc:460] Webserver started at http://127.0.192.254:39915/ using document root <none> and password file <none>
I20260621 07:47:37.232172 6315 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:37.232250 6315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:37.232439 6315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:37.234094 6315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "16856977edf5464e809c5af8d2d6a493"
format_stamp: "Formatted at 2026-06-21 07:47:37 on dist-test-slave-lzx3"
server_key: "e071f1e9c82fb8091ef7eb3199546aaf"
server_key_iv: "aef7ab16bd4971893bbfa5ce52089c5e"
server_key_version: "encryptionkey@0"
I20260621 07:47:37.234691 6315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "16856977edf5464e809c5af8d2d6a493"
format_stamp: "Formatted at 2026-06-21 07:47:37 on dist-test-slave-lzx3"
server_key: "e071f1e9c82fb8091ef7eb3199546aaf"
server_key_iv: "aef7ab16bd4971893bbfa5ce52089c5e"
server_key_version: "encryptionkey@0"
I20260621 07:47:37.238104 6315 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 07:47:37.240046 6330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:37.241158 6315 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:37.241273 6315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "16856977edf5464e809c5af8d2d6a493"
format_stamp: "Formatted at 2026-06-21 07:47:37 on dist-test-slave-lzx3"
server_key: "e071f1e9c82fb8091ef7eb3199546aaf"
server_key_iv: "aef7ab16bd4971893bbfa5ce52089c5e"
server_key_version: "encryptionkey@0"
I20260621 07:47:37.241359 6315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:37.252683 6315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:37.254822 6315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:37.254994 6315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:37.260883 6315 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:45857
I20260621 07:47:37.261098 6382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:45857 every 8 connection(s)
I20260621 07:47:37.261624 6315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:47:37.263918 6383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:37.268010 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 6315
I20260621 07:47:37.268139 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:47:37.268378 771 external_mini_cluster.cc:1428] Setting key ca5bdbc3e205922334ddc11bb37e4085
I20260621 07:47:37.268631 6383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16856977edf5464e809c5af8d2d6a493: Bootstrap starting.
I20260621 07:47:37.270598 6383 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16856977edf5464e809c5af8d2d6a493: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:37.271366 6383 log.cc:824] T 00000000000000000000000000000000 P 16856977edf5464e809c5af8d2d6a493: Log is configured to *not* fsync() on all Append() calls
I20260621 07:47:37.272878 6383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16856977edf5464e809c5af8d2d6a493: No bootstrap required, opened a new log
Jun 21 07:47:37 dist-test-slave-lzx3 krb5kdc[6302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:47:37.276285 6383 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16856977edf5464e809c5af8d2d6a493 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16856977edf5464e809c5af8d2d6a493" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45857 } }
I20260621 07:47:37.276461 6383 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16856977edf5464e809c5af8d2d6a493 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:37.276510 6383 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16856977edf5464e809c5af8d2d6a493 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16856977edf5464e809c5af8d2d6a493, State: Initialized, Role: FOLLOWER
I20260621 07:47:37.276973 6383 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16856977edf5464e809c5af8d2d6a493 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16856977edf5464e809c5af8d2d6a493" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45857 } }
I20260621 07:47:37.277137 6383 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16856977edf5464e809c5af8d2d6a493 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:47:37.277213 6383 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16856977edf5464e809c5af8d2d6a493 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:47:37.277313 6383 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16856977edf5464e809c5af8d2d6a493 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:37.278071 6383 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16856977edf5464e809c5af8d2d6a493 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16856977edf5464e809c5af8d2d6a493" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45857 } }
I20260621 07:47:37.278383 6383 leader_election.cc:302] T 00000000000000000000000000000000 P 16856977edf5464e809c5af8d2d6a493 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16856977edf5464e809c5af8d2d6a493; no voters:
I20260621 07:47:37.278693 6383 leader_election.cc:288] T 00000000000000000000000000000000 P 16856977edf5464e809c5af8d2d6a493 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:47:37.278779 6388 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16856977edf5464e809c5af8d2d6a493 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:47:37.279007 6388 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16856977edf5464e809c5af8d2d6a493 [term 1 LEADER]: Becoming Leader. State: Replica: 16856977edf5464e809c5af8d2d6a493, State: Running, Role: LEADER
I20260621 07:47:37.279301 6388 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16856977edf5464e809c5af8d2d6a493 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16856977edf5464e809c5af8d2d6a493" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45857 } }
I20260621 07:47:37.279655 6383 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16856977edf5464e809c5af8d2d6a493 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:47:37.280290 6390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16856977edf5464e809c5af8d2d6a493 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16856977edf5464e809c5af8d2d6a493. Latest consensus state: current_term: 1 leader_uuid: "16856977edf5464e809c5af8d2d6a493" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16856977edf5464e809c5af8d2d6a493" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45857 } } }
I20260621 07:47:37.280414 6390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16856977edf5464e809c5af8d2d6a493 [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:37.281051 6389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16856977edf5464e809c5af8d2d6a493 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16856977edf5464e809c5af8d2d6a493" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16856977edf5464e809c5af8d2d6a493" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45857 } } }
I20260621 07:47:37.281170 6386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:37.269584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47536 (local address 127.0.192.254:45857)
0621 07:47:37.269950 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:37.269965 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:37.269993 (+ 28us) server_negotiation.cc:407] Connection header received
0621 07:47:37.270475 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:37.270487 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:37.270773 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:37.271014 (+ 241us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:37.271783 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:37.272591 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:37.273384 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:37.273766 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:37.273912 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:37.276325 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:37.276329 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:37.276338 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:37.276359 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:37.278037 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:37.278452 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:37.278456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:37.278459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:37.278580 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:37.278831 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:37.278837 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:37.278839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:37.279266 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:37.279437 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:37.280008 (+ 571us) server_negotiation.cc:299] Negotiation successful
0621 07:47:37.280259 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":125,"threads_started":1}
I20260621 07:47:37.281184 6389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16856977edf5464e809c5af8d2d6a493 [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:37.281876 6397 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:47:37.284453 6397 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:47:37.289149 6397 catalog_manager.cc:1347] Generated new cluster ID: f53a1d45e10d4e5ba84aa4dd8981b671
I20260621 07:47:37.289206 6397 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:47:37.302351 6397 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:47:37.303205 6397 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:47:37.306667 6397 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16856977edf5464e809c5af8d2d6a493: Generated new TSK 0
I20260621 07:47:37.307226 6397 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.192.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab.
Entry for principal kudu/127.0.192.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab.
I20260621 07:47:37.332829 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.193:0
--local_ip_for_outbound_sockets=127.0.192.193
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:45857
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:36851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:37.426942 6409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:37.427173 6409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:37.427237 6409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:37.430687 6409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:37.430866 6409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:37.430981 6409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.193
I20260621 07:47:37.435664 6409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:36851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:45857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.193
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:37.436425 6409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:37.437296 6409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:37.445392 6414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:37.445415 6417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:37.445415 6415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:37.445477 6409 server_base.cc:1034] running on GCE node
I20260621 07:47:37.445926 6409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:37.446435 6409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:37.447610 6409 hybrid_clock.cc:648] HybridClock initialized: now 1782028057447598 us; error 30 us; skew 500 ppm
Jun 21 07:47:37 dist-test-slave-lzx3 krb5kdc[6302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:37.450001 6409 init.cc:377] Logged in from keytab as kudu/127.0.192.193@KRBTEST.COM (short username kudu)
I20260621 07:47:37.450660 6409 webserver.cc:460] Webserver started at http://127.0.192.193:42235/ using document root <none> and password file <none>
I20260621 07:47:37.451095 6409 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:37.451169 6409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:37.451349 6409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:37.452998 6409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-0/data/instance:
uuid: "1756f2d4b37a492b9f59e3c413576d69"
format_stamp: "Formatted at 2026-06-21 07:47:37 on dist-test-slave-lzx3"
server_key: "910a9d4d146aff1e1617523dfc0be8a0"
server_key_iv: "4bf0f09d5f557de2d374b21134ebe627"
server_key_version: "encryptionkey@0"
I20260621 07:47:37.453464 6409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-0/wal/instance:
uuid: "1756f2d4b37a492b9f59e3c413576d69"
format_stamp: "Formatted at 2026-06-21 07:47:37 on dist-test-slave-lzx3"
server_key: "910a9d4d146aff1e1617523dfc0be8a0"
server_key_iv: "4bf0f09d5f557de2d374b21134ebe627"
server_key_version: "encryptionkey@0"
I20260621 07:47:37.456488 6409 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:47:37.458550 6424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:37.459604 6409 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:37.459748 6409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-0/wal
uuid: "1756f2d4b37a492b9f59e3c413576d69"
format_stamp: "Formatted at 2026-06-21 07:47:37 on dist-test-slave-lzx3"
server_key: "910a9d4d146aff1e1617523dfc0be8a0"
server_key_iv: "4bf0f09d5f557de2d374b21134ebe627"
server_key_version: "encryptionkey@0"
I20260621 07:47:37.459837 6409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:37.477744 6409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:37.479974 6409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:37.480121 6409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:37.480525 6409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:47:37.481213 6409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:47:37.481254 6409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:37.481287 6409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:47:37.481304 6409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:37.488663 6409 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.193:45175
I20260621 07:47:37.488690 6537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.193:45175 every 8 connection(s)
I20260621 07:47:37.489355 6409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
Jun 21 07:47:37 dist-test-slave-lzx3 krb5kdc[6302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:47:37.497640 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 6409
I20260621 07:47:37.497757 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-0/wal/instance
I20260621 07:47:37.498090 771 external_mini_cluster.cc:1428] Setting key bb20b7673e40d5343c3d7817d621c28a
I20260621 07:47:37.500881 6386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:37.490714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:39879 (local address 127.0.192.254:45857)
0621 07:47:37.490825 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:37.490830 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:37.491439 (+ 609us) server_negotiation.cc:407] Connection header received
0621 07:47:37.492662 (+ 1223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:37.492666 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:37.492721 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:37.492829 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:37.493740 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:37.494199 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:37.495248 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:37.495478 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:37.495583 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:37.497565 (+ 1982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:37.497568 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:37.497570 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:37.497592 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:37.498995 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:37.499492 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:37.499495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:37.499496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:37.499558 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:37.499881 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:37.499882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:37.499883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:37.500014 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:37.500076 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:37.500607 (+ 531us) server_negotiation.cc:299] Negotiation successful
0621 07:47:37.500720 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260621 07:47:37.501916 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:37.490964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45857 (local address 127.0.192.193:39879)
0621 07:47:37.491326 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:37.491357 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:37.492451 (+ 1094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:37.492969 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:37.492974 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:37.493259 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:37.493607 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:37.493617 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:37.494306 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:37.494309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:37.495126 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:37.495134 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:37.495620 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:37.495623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:37.495767 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:37.496160 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:37.496172 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:37.497383 (+ 1211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:37.499095 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:37.499099 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:37.499107 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:37.499405 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:37.499639 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:37.499642 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:37.499643 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:37.499795 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:37.500096 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:37.500104 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:37.500358 (+ 254us) client_negotiation.cc:769] Sending connection context
0621 07:47:37.500524 (+ 166us) client_negotiation.cc:240] Negotiation successful
0621 07:47:37.500898 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":100,"threads_started":1}
I20260621 07:47:37.502936 6538 heartbeater.cc:347] Connected to a master server at 127.0.192.254:45857
I20260621 07:47:37.503124 6538 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:37.503538 6538 heartbeater.cc:511] Master 127.0.192.254:45857 requested a full tablet report, sending...
I20260621 07:47:37.504997 6347 ts_manager.cc:194] Registered new tserver with Master: 1756f2d4b37a492b9f59e3c413576d69 (127.0.192.193:45175)
I20260621 07:47:37.506394 6347 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.193@KRBTEST.COM'} at 127.0.192.193:39879
WARNING: no policy specified for kudu/127.0.192.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab.
Entry for principal kudu/127.0.192.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab.
I20260621 07:47:37.522682 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.194:0
--local_ip_for_outbound_sockets=127.0.192.194
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:45857
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:36851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:37.619625 6543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:37.619863 6543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:37.619932 6543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:37.623350 6543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:37.623417 6543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:37.623494 6543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.194
I20260621 07:47:37.627882 6543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:36851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:45857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.194
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:37.628721 6543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:37.629518 6543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:37.637614 6548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:37.637621 6551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:37.637727 6543 server_base.cc:1034] running on GCE node
W20260621 07:47:37.637637 6549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:37.638254 6543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:37.638741 6543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:37.639904 6543 hybrid_clock.cc:648] HybridClock initialized: now 1782028057639890 us; error 20 us; skew 500 ppm
Jun 21 07:47:37 dist-test-slave-lzx3 krb5kdc[6302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:37.642230 6543 init.cc:377] Logged in from keytab as kudu/127.0.192.194@KRBTEST.COM (short username kudu)
I20260621 07:47:37.642895 6543 webserver.cc:460] Webserver started at http://127.0.192.194:37025/ using document root <none> and password file <none>
I20260621 07:47:37.643337 6543 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:37.643411 6543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:37.643591 6543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:37.645390 6543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-1/data/instance:
uuid: "41bf8fe0dd5740748f2a081e48fef66d"
format_stamp: "Formatted at 2026-06-21 07:47:37 on dist-test-slave-lzx3"
server_key: "53526b451ca0efe87d7358b03e5d8d16"
server_key_iv: "6306572c5defe1828524a2a71cd22b89"
server_key_version: "encryptionkey@0"
I20260621 07:47:37.645812 6543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-1/wal/instance:
uuid: "41bf8fe0dd5740748f2a081e48fef66d"
format_stamp: "Formatted at 2026-06-21 07:47:37 on dist-test-slave-lzx3"
server_key: "53526b451ca0efe87d7358b03e5d8d16"
server_key_iv: "6306572c5defe1828524a2a71cd22b89"
server_key_version: "encryptionkey@0"
I20260621 07:47:37.648959 6543 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:47:37.651084 6558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:37.652163 6543 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:37.652281 6543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-1/wal
uuid: "41bf8fe0dd5740748f2a081e48fef66d"
format_stamp: "Formatted at 2026-06-21 07:47:37 on dist-test-slave-lzx3"
server_key: "53526b451ca0efe87d7358b03e5d8d16"
server_key_iv: "6306572c5defe1828524a2a71cd22b89"
server_key_version: "encryptionkey@0"
I20260621 07:47:37.652388 6543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:37.663244 6543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:37.665719 6543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:37.665887 6543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:37.666365 6543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:47:37.667106 6543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:47:37.667177 6543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:37.667241 6543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:47:37.667287 6543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:37.676379 6543 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.194:42581
I20260621 07:47:37.676434 6671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.194:42581 every 8 connection(s)
I20260621 07:47:37.677394 6543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
Jun 21 07:47:37 dist-test-slave-lzx3 krb5kdc[6302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:47:37.687105 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 6543
I20260621 07:47:37.687248 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-1/wal/instance
I20260621 07:47:37.687500 771 external_mini_cluster.cc:1428] Setting key 7978416f368ac5c25759729a1477a73c
I20260621 07:47:37.689215 6386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:37.679418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:50393 (local address 127.0.192.254:45857)
0621 07:47:37.679522 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:37.679526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:37.680213 (+ 687us) server_negotiation.cc:407] Connection header received
0621 07:47:37.681455 (+ 1242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:37.681458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:37.681532 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:37.681607 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:37.682481 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:37.682912 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:37.683817 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:37.683999 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:37.684064 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:37.686053 (+ 1989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:37.686056 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:37.686058 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:37.686079 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:37.687286 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:37.687809 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:37.687813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:37.687815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:37.687896 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:37.688226 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:37.688229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:37.688231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:37.688384 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:37.688493 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:37.688953 (+ 460us) server_negotiation.cc:299] Negotiation successful
0621 07:47:37.689055 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260621 07:47:37.689882 6674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:37.679727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45857 (local address 127.0.192.194:50393)
0621 07:47:37.680083 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:37.680118 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:37.681291 (+ 1173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:37.681744 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:37.681750 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:37.682018 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:37.682378 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:37.682389 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:37.682996 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:37.682998 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:37.683729 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:37.683737 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:37.684101 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:37.684103 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:37.684221 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:37.684683 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:37.684697 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:37.685916 (+ 1219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:37.687392 (+ 1476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:37.687397 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:37.687409 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:37.687711 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:37.687993 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:37.687996 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:37.687998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:37.688136 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:37.688473 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:37.688495 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:37.688741 (+ 246us) client_negotiation.cc:769] Sending connection context
0621 07:47:37.688900 (+ 159us) client_negotiation.cc:240] Negotiation successful
0621 07:47:37.689155 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":90,"threads_started":1}
I20260621 07:47:37.691010 6672 heartbeater.cc:347] Connected to a master server at 127.0.192.254:45857
I20260621 07:47:37.691231 6672 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:37.691679 6672 heartbeater.cc:511] Master 127.0.192.254:45857 requested a full tablet report, sending...
I20260621 07:47:37.692653 6347 ts_manager.cc:194] Registered new tserver with Master: 41bf8fe0dd5740748f2a081e48fef66d (127.0.192.194:42581)
I20260621 07:47:37.693135 6347 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.194@KRBTEST.COM'} at 127.0.192.194:50393
WARNING: no policy specified for kudu/127.0.192.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab.
Entry for principal kudu/127.0.192.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab.
I20260621 07:47:37.710916 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.195:0
--local_ip_for_outbound_sockets=127.0.192.195
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:45857
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:36851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:37.805714 6677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:37.805912 6677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:37.805946 6677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:37.809818 6677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:37.809871 6677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:37.809922 6677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.195
I20260621 07:47:37.814234 6677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:36851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:45857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.195
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:37.814996 6677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:37.815791 6677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:37.823978 6682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:37.823982 6685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:37.824054 6683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:37.824163 6677 server_base.cc:1034] running on GCE node
I20260621 07:47:37.824698 6677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:37.825201 6677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:37.826364 6677 hybrid_clock.cc:648] HybridClock initialized: now 1782028057826354 us; error 25 us; skew 500 ppm
Jun 21 07:47:37 dist-test-slave-lzx3 krb5kdc[6302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:37.828896 6677 init.cc:377] Logged in from keytab as kudu/127.0.192.195@KRBTEST.COM (short username kudu)
I20260621 07:47:37.829586 6677 webserver.cc:460] Webserver started at http://127.0.192.195:39707/ using document root <none> and password file <none>
I20260621 07:47:37.830044 6677 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:37.830097 6677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:37.830264 6677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:37.831797 6677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-2/data/instance:
uuid: "80ef4328dadb41aaba609829c379d28c"
format_stamp: "Formatted at 2026-06-21 07:47:37 on dist-test-slave-lzx3"
server_key: "9fea599ce906680d4f2dec8413449553"
server_key_iv: "0c3d4708c31a07ca2bbd75ce1cb5fc71"
server_key_version: "encryptionkey@0"
I20260621 07:47:37.832235 6677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-2/wal/instance:
uuid: "80ef4328dadb41aaba609829c379d28c"
format_stamp: "Formatted at 2026-06-21 07:47:37 on dist-test-slave-lzx3"
server_key: "9fea599ce906680d4f2dec8413449553"
server_key_iv: "0c3d4708c31a07ca2bbd75ce1cb5fc71"
server_key_version: "encryptionkey@0"
I20260621 07:47:37.835459 6677 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 07:47:37.837473 6692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:37.838508 6677 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:37.838603 6677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-2/wal
uuid: "80ef4328dadb41aaba609829c379d28c"
format_stamp: "Formatted at 2026-06-21 07:47:37 on dist-test-slave-lzx3"
server_key: "9fea599ce906680d4f2dec8413449553"
server_key_iv: "0c3d4708c31a07ca2bbd75ce1cb5fc71"
server_key_version: "encryptionkey@0"
I20260621 07:47:37.838670 6677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:37.865823 6677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:37.868618 6677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:37.868767 6677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:37.869179 6677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:47:37.869786 6677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:47:37.869823 6677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:37.869853 6677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:47:37.869870 6677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:37.877338 6677 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.195:33001
I20260621 07:47:37.877363 6805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.195:33001 every 8 connection(s)
I20260621 07:47:37.878074 6677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
I20260621 07:47:37.885281 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 6677
I20260621 07:47:37.885396 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/minicluster-data/ts-2/wal/instance
I20260621 07:47:37.885641 771 external_mini_cluster.cc:1428] Setting key b5c073b6c32c42276507c6ae396ebf79
Jun 21 07:47:37 dist-test-slave-lzx3 krb5kdc[6302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:47:37.889300 6386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:37.879522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:51601 (local address 127.0.192.254:45857)
0621 07:47:37.879630 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:37.879635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:37.880399 (+ 764us) server_negotiation.cc:407] Connection header received
0621 07:47:37.881590 (+ 1191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:37.881593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:37.881635 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:37.881700 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:37.882555 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:37.883002 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:37.883928 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:37.884108 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:37.884180 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:37.886333 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:37.886338 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:37.886342 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:37.886366 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:37.887823 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:37.888232 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:37.888234 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:37.888235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:37.888293 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:37.888519 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:37.888521 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:37.888522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:37.888712 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:37.888777 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:37.889127 (+ 350us) server_negotiation.cc:299] Negotiation successful
0621 07:47:37.889202 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260621 07:47:37.889957 6808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:37.879878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45857 (local address 127.0.192.195:51601)
0621 07:47:37.880278 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:37.880311 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:37.881429 (+ 1118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:37.881817 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:37.881822 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:37.882082 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:37.882452 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:37.882462 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:37.883094 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:37.883096 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:37.883836 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:37.883844 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:37.884220 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:37.884222 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:37.884337 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:37.884789 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:37.884803 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:37.886194 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:37.887911 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:37.887915 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:37.887925 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:37.888158 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:37.888354 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:37.888356 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:37.888357 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:37.888456 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:37.888775 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:37.888778 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:37.888944 (+ 166us) client_negotiation.cc:769] Sending connection context
0621 07:47:37.889091 (+ 147us) client_negotiation.cc:240] Negotiation successful
0621 07:47:37.889280 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":101,"threads_started":1}
I20260621 07:47:37.890816 6806 heartbeater.cc:347] Connected to a master server at 127.0.192.254:45857
I20260621 07:47:37.891008 6806 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:37.891386 6806 heartbeater.cc:511] Master 127.0.192.254:45857 requested a full tablet report, sending...
I20260621 07:47:37.892176 6347 ts_manager.cc:194] Registered new tserver with Master: 80ef4328dadb41aaba609829c379d28c (127.0.192.195:33001)
I20260621 07:47:37.892647 6347 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.195@KRBTEST.COM'} at 127.0.192.195:51601
I20260621 07:47:37.898969 771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/krb5kdc/test-admin.keytab.
Jun 21 07:47:37 dist-test-slave-lzx3 krb5kdc[6302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:37.912261 771 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260621 07:47:37.914336 6813 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:37.913295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38212 (local address 127.0.192.193:45175)
0621 07:47:37.913542 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:37.913555 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:37.913574 (+ 19us) server_negotiation.cc:407] Connection header received
0621 07:47:37.913611 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:37.913614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:37.913749 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:37.913845 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:37.914147 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38212: BlockingRecv error: recv got EOF from 127.0.0.1:38212 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":94,"threads_started":1}
W20260621 07:47:37.914489 771 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/krb5kdc/krb5cc' of type FILE
Jun 21 07:47:37 dist-test-slave-lzx3 krb5kdc[6302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:37.915395 771 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 21 07:47:37 dist-test-slave-lzx3 krb5kdc[6302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:37.916188 771 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 21 07:47:37 dist-test-slave-lzx3 krb5kdc[6302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.192.193@KRBTEST.COM
I20260621 07:47:37.923844 6813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:37.916844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38226 (local address 127.0.192.193:45175)
0621 07:47:37.916977 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:37.916981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:37.917068 (+ 87us) server_negotiation.cc:407] Connection header received
0621 07:47:37.917151 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:37.917153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:37.917186 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:37.917250 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:37.917695 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:37.918364 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:37.919106 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:37.919387 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:37.919459 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:37.921265 (+ 1806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:37.921273 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:37.921281 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:37.921299 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:37.922571 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:37.922900 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:37.922904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:37.922905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:37.922967 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:37.923145 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:37.923147 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:37.923147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:37.923315 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:37.923443 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:37.923556 (+ 113us) server_negotiation.cc:299] Negotiation successful
0621 07:47:37.923705 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260621 07:47:37.926959 6813 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:37.926053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38230 (local address 127.0.192.193:45175)
0621 07:47:37.926153 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:37.926156 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:37.926238 (+ 82us) server_negotiation.cc:407] Connection header received
0621 07:47:37.926349 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:37.926352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:37.926381 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:37.926454 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:37.926832 (+ 378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38230: BlockingRecv error: recv got EOF from 127.0.0.1:38230 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260621 07:47:37.927080 771 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/krb5kdc/krb5cc' of type FILE
Jun 21 07:47:37 dist-test-slave-lzx3 krb5kdc[6302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:37.927963 771 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 21 07:47:37 dist-test-slave-lzx3 krb5kdc[6302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:37.928757 771 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 21 07:47:37 dist-test-slave-lzx3 krb5kdc[6302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.192.193@KRBTEST.COM
I20260621 07:47:37.935743 6813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:37.929374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38244 (local address 127.0.192.193:45175)
0621 07:47:37.929468 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:37.929472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:37.929549 (+ 77us) server_negotiation.cc:407] Connection header received
0621 07:47:37.929647 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:37.929651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:37.929681 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:37.929763 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:37.930179 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:37.930590 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:37.931256 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:37.931456 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:37.931546 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:37.933250 (+ 1704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:37.933253 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:37.933255 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:37.933275 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:37.934257 (+ 982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:37.934655 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:37.934660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:37.934662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:37.934744 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:37.935017 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:37.935020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:37.935022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:37.935161 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:37.935277 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:37.935498 (+ 221us) server_negotiation.cc:299] Negotiation successful
0621 07:47:37.935607 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260621 07:47:37.937811 6813 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:37.937000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38254 (local address 127.0.192.193:45175)
0621 07:47:37.937114 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:37.937118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:37.937162 (+ 44us) server_negotiation.cc:407] Connection header received
0621 07:47:37.937251 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:37.937253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:37.937281 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:37.937339 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:37.937722 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38254: BlockingRecv error: recv got EOF from 127.0.0.1:38254 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260621 07:47:37.937811 771 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782028028522580-771-0/krb5kdc/krb5cc' of type FILE
Jun 21 07:47:37 dist-test-slave-lzx3 krb5kdc[6302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:37.938657 771 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 21 07:47:37 dist-test-slave-lzx3 krb5kdc[6302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:37.939426 771 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 21 07:47:37 dist-test-slave-lzx3 krb5kdc[6302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.192.193@KRBTEST.COM
I20260621 07:47:37.946540 6813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:37.940091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38270 (local address 127.0.192.193:45175)
0621 07:47:37.940231 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:37.940235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:37.940334 (+ 99us) server_negotiation.cc:407] Connection header received
0621 07:47:37.940455 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:37.940470 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:37.940500 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:37.940583 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:37.941022 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:37.941435 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:37.942124 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:37.942314 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:37.942397 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:37.944225 (+ 1828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:37.944228 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:37.944230 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:37.944253 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:37.945303 (+ 1050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:37.945632 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:37.945636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:37.945638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:37.945702 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:37.945931 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:37.945933 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:37.945935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:37.946054 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:37.946164 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:37.946351 (+ 187us) server_negotiation.cc:299] Negotiation successful
0621 07:47:37.946438 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 07:47:37.947384 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 6409
I20260621 07:47:37.952804 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 6543
I20260621 07:47:37.957602 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 6677
I20260621 07:47:37.962110 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 6315
2026-06-21T07:47:37Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1906 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:47:38 dist-test-slave-lzx3 krb5kdc[6826](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:47:38 dist-test-slave-lzx3 krb5kdc[6826](info): set up 2 sockets
Jun 21 07:47:38 dist-test-slave-lzx3 krb5kdc[6826](info): commencing operation
krb5kdc: starting...
W20260621 07:47:38.904455 771 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/test-user.keytab.
Jun 21 07:47:38 dist-test-slave-lzx3 krb5kdc[6826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:47:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:47:38Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.192.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.192.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.192.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/oryx_127.0.192.254.keytab.
Entry for principal oryx/127.0.192.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/oryx_127.0.192.254.keytab.
I20260621 07:47:39.008595 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:40701
--webserver_interface=127.0.192.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/oryx_127.0.192.254.keytab
--principal=oryx/127.0.192.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:39257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:40701
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:39.106299 6839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:39.106526 6839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:39.106591 6839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:39.110337 6839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:47:39.110409 6839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:39.110452 6839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:39.110502 6839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:47:39.110539 6839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:47:39.115295 6839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:39257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:40701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/oryx_127.0.192.254.keytab
--principal=oryx/127.0.192.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:40701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:39.116146 6839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:39.117007 6839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:39.124447 6847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:39.124480 6844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:39.124456 6845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:39.124464 6839 server_base.cc:1034] running on GCE node
I20260621 07:47:39.125159 6839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:39.125977 6839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:39.127146 6839 hybrid_clock.cc:648] HybridClock initialized: now 1782028059127125 us; error 35 us; skew 500 ppm
Jun 21 07:47:39 dist-test-slave-lzx3 krb5kdc[6826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028059, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.192.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:39.129642 6839 init.cc:377] Logged in from keytab as oryx/127.0.192.254@KRBTEST.COM (short username oryx)
I20260621 07:47:39.130331 6839 webserver.cc:460] Webserver started at http://127.0.192.254:44551/ using document root <none> and password file <none>
I20260621 07:47:39.130777 6839 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:39.130848 6839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:39.131027 6839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:39.132637 6839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "c8072ff3594143ca8e36eb920eae693c"
format_stamp: "Formatted at 2026-06-21 07:47:39 on dist-test-slave-lzx3"
server_key: "993e59880aded8af307da3924e1ac85d"
server_key_iv: "d6e58fc6d32d970ed241967e8b887000"
server_key_version: "encryptionkey@0"
I20260621 07:47:39.133114 6839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "c8072ff3594143ca8e36eb920eae693c"
format_stamp: "Formatted at 2026-06-21 07:47:39 on dist-test-slave-lzx3"
server_key: "993e59880aded8af307da3924e1ac85d"
server_key_iv: "d6e58fc6d32d970ed241967e8b887000"
server_key_version: "encryptionkey@0"
I20260621 07:47:39.136152 6839 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260621 07:47:39.138200 6854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:39.139222 6839 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 07:47:39.139326 6839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "c8072ff3594143ca8e36eb920eae693c"
format_stamp: "Formatted at 2026-06-21 07:47:39 on dist-test-slave-lzx3"
server_key: "993e59880aded8af307da3924e1ac85d"
server_key_iv: "d6e58fc6d32d970ed241967e8b887000"
server_key_version: "encryptionkey@0"
I20260621 07:47:39.139415 6839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:39.153142 6839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:39.155413 6839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:39.155581 6839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:39.161471 6839 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:40701
I20260621 07:47:39.161501 6906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:40701 every 8 connection(s)
I20260621 07:47:39.162197 6839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:47:39.163147 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 6839
I20260621 07:47:39.163272 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:47:39.163542 771 external_mini_cluster.cc:1428] Setting key b31473a220f4f2851a5789b86430e277
I20260621 07:47:39.164934 6907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:39.170284 6907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c: Bootstrap starting.
Jun 21 07:47:39 dist-test-slave-lzx3 krb5kdc[6826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM
I20260621 07:47:39.172221 6907 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:39.172851 6907 log.cc:824] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c: Log is configured to *not* fsync() on all Append() calls
I20260621 07:47:39.174636 6907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c: No bootstrap required, opened a new log
I20260621 07:47:39.175290 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:39.164654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45174 (local address 127.0.192.254:40701)
0621 07:47:39.164984 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:39.165004 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:39.165049 (+ 45us) server_negotiation.cc:407] Connection header received
0621 07:47:39.165545 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:39.165558 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:39.165806 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:39.166029 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:39.166769 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:39.167663 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:39.168838 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:39.169199 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:39.169296 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:39.171493 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:39.171497 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:39.171506 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:39.171527 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:39.172923 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:39.173347 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:39.173352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:39.173357 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:39.173472 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:39.173725 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:39.173727 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:39.173728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:39.174020 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:39.174150 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:39.174389 (+ 239us) server_negotiation.cc:299] Negotiation successful
0621 07:47:39.174662 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":125,"threads_started":1}
I20260621 07:47:39.177915 6907 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } }
I20260621 07:47:39.178121 6907 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:39.178200 6907 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8072ff3594143ca8e36eb920eae693c, State: Initialized, Role: FOLLOWER
I20260621 07:47:39.178548 6907 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } }
I20260621 07:47:39.178692 6907 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:47:39.178766 6907 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:47:39.178860 6907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:39.179538 6907 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } }
I20260621 07:47:39.179831 6907 leader_election.cc:302] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8072ff3594143ca8e36eb920eae693c; no voters:
I20260621 07:47:39.180091 6907 leader_election.cc:288] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:47:39.180186 6912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:47:39.180624 6912 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [term 1 LEADER]: Becoming Leader. State: Replica: c8072ff3594143ca8e36eb920eae693c, State: Running, Role: LEADER
I20260621 07:47:39.180868 6912 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } }
I20260621 07:47:39.181085 6907 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:47:39.182296 6913 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8072ff3594143ca8e36eb920eae693c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } }
I20260621 07:47:39.182447 6913 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:39.182292 6914 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8072ff3594143ca8e36eb920eae693c. Latest consensus state: current_term: 1 leader_uuid: "c8072ff3594143ca8e36eb920eae693c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } }
I20260621 07:47:39.182643 6914 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:39.182760 6925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:47:39.185341 6925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:47:39.190290 6925 catalog_manager.cc:1347] Generated new cluster ID: 2feba99244ef42a5b71acfd2c5120240
I20260621 07:47:39.190335 6925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:47:39.194226 6925 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:47:39.195245 6925 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:47:39.201531 6925 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c: Generated new TSK 0
I20260621 07:47:39.202046 6925 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.192.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.192.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.192.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/oryx_127.0.192.193.keytab.
Entry for principal oryx/127.0.192.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/oryx_127.0.192.193.keytab.
I20260621 07:47:39.230443 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.193:0
--local_ip_for_outbound_sockets=127.0.192.193
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:40701
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/oryx_127.0.192.193.keytab
--principal=oryx/127.0.192.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:39257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:39.325518 6933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:39.325730 6933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:39.325789 6933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:39.329444 6933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:39.329502 6933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:39.329572 6933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.193
I20260621 07:47:39.333765 6933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:39257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/oryx_127.0.192.193.keytab
--principal=oryx/127.0.192.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:40701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.193
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:39.334517 6933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:39.335377 6933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:39.343205 6941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:39.343273 6938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:39.343353 6933 server_base.cc:1034] running on GCE node
W20260621 07:47:39.343262 6939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:39.343696 6933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:39.344167 6933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:39.345304 6933 hybrid_clock.cc:648] HybridClock initialized: now 1782028059345290 us; error 24 us; skew 500 ppm
Jun 21 07:47:39 dist-test-slave-lzx3 krb5kdc[6826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028059, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.192.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:39.347644 6933 init.cc:377] Logged in from keytab as oryx/127.0.192.193@KRBTEST.COM (short username oryx)
I20260621 07:47:39.348387 6933 webserver.cc:460] Webserver started at http://127.0.192.193:39811/ using document root <none> and password file <none>
I20260621 07:47:39.348848 6933 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:39.348898 6933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:39.349089 6933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:39.350615 6933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-0/data/instance:
uuid: "efd8a37d6ac44bb4b4ef2f0e61cdb11c"
format_stamp: "Formatted at 2026-06-21 07:47:39 on dist-test-slave-lzx3"
server_key: "02e7d2242d7c16d9478f24900dda7d17"
server_key_iv: "9694162935e13f6ddc90bb5cb9c71609"
server_key_version: "encryptionkey@0"
I20260621 07:47:39.351068 6933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-0/wal/instance:
uuid: "efd8a37d6ac44bb4b4ef2f0e61cdb11c"
format_stamp: "Formatted at 2026-06-21 07:47:39 on dist-test-slave-lzx3"
server_key: "02e7d2242d7c16d9478f24900dda7d17"
server_key_iv: "9694162935e13f6ddc90bb5cb9c71609"
server_key_version: "encryptionkey@0"
I20260621 07:47:39.353996 6933 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 07:47:39.355963 6948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:39.356962 6933 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:39.357069 6933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-0/wal
uuid: "efd8a37d6ac44bb4b4ef2f0e61cdb11c"
format_stamp: "Formatted at 2026-06-21 07:47:39 on dist-test-slave-lzx3"
server_key: "02e7d2242d7c16d9478f24900dda7d17"
server_key_iv: "9694162935e13f6ddc90bb5cb9c71609"
server_key_version: "encryptionkey@0"
I20260621 07:47:39.357190 6933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:39.384434 6933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:39.386924 6933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:39.387079 6933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:39.387586 6933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:47:39.388290 6933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:47:39.388372 6933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:39.388428 6933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:47:39.388468 6933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:39.395998 6933 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.193:45293
I20260621 07:47:39.396018 7061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.193:45293 every 8 connection(s)
I20260621 07:47:39.396687 6933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
Jun 21 07:47:39 dist-test-slave-lzx3 krb5kdc[6826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028059, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.192.193@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM
I20260621 07:47:39.405241 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 6933
I20260621 07:47:39.405371 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-0/wal/instance
I20260621 07:47:39.405673 771 external_mini_cluster.cc:1428] Setting key 28cdf80e07563cf36da50eba27f0573d
I20260621 07:47:39.407872 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:39.398162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:60269 (local address 127.0.192.254:40701)
0621 07:47:39.398313 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:39.398318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:39.398991 (+ 673us) server_negotiation.cc:407] Connection header received
0621 07:47:39.399996 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:39.399998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:39.400038 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:39.400102 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:39.401038 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:39.401480 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:39.402352 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:39.402536 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:39.402600 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:39.404463 (+ 1863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:39.404465 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:39.404467 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:39.404486 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:39.405920 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:39.406401 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:39.406404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:39.406405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:39.406479 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:39.406846 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:39.406849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:39.406850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:39.407029 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:39.407092 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:39.407603 (+ 511us) server_negotiation.cc:299] Negotiation successful
0621 07:47:39.407716 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 07:47:39.408501 7064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:39.398508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:40701 (local address 127.0.192.193:60269)
0621 07:47:39.398865 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:39.398898 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:39.399836 (+ 938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:39.400222 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:39.400227 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:39.400489 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:39.400916 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:39.400927 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:39.401565 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:39.401567 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:39.402264 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:39.402271 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:39.402635 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:39.402638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:39.402744 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:39.403204 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:39.403216 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:39.404335 (+ 1119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:39.406029 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:39.406034 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:39.406045 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:39.406312 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:39.406586 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:39.406588 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:39.406590 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:39.406732 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:39.407194 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:39.407201 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:39.407393 (+ 192us) client_negotiation.cc:769] Sending connection context
0621 07:47:39.407560 (+ 167us) client_negotiation.cc:240] Negotiation successful
0621 07:47:39.407821 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":95,"threads_started":1}
I20260621 07:47:39.409595 7062 heartbeater.cc:347] Connected to a master server at 127.0.192.254:40701
I20260621 07:47:39.409767 7062 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:39.410122 7062 heartbeater.cc:511] Master 127.0.192.254:40701 requested a full tablet report, sending...
I20260621 07:47:39.411254 6871 ts_manager.cc:194] Registered new tserver with Master: efd8a37d6ac44bb4b4ef2f0e61cdb11c (127.0.192.193:45293)
I20260621 07:47:39.412333 6871 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.192.193@KRBTEST.COM'} at 127.0.192.193:60269
WARNING: no policy specified for oryx/127.0.192.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.192.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.192.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/oryx_127.0.192.194.keytab.
Entry for principal oryx/127.0.192.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/oryx_127.0.192.194.keytab.
I20260621 07:47:39.428948 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.194:0
--local_ip_for_outbound_sockets=127.0.192.194
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:40701
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/oryx_127.0.192.194.keytab
--principal=oryx/127.0.192.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:39257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:39.522149 7067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:39.522332 7067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:39.522398 7067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:39.525903 7067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:39.525945 7067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:39.526031 7067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.194
I20260621 07:47:39.530503 7067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:39257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/oryx_127.0.192.194.keytab
--principal=oryx/127.0.192.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:40701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.194
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:39.531240 7067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:39.532045 7067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:39.539889 7072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:39.539893 7073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:39.540202 7067 server_base.cc:1034] running on GCE node
W20260621 07:47:39.539889 7075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:39.540793 7067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:39.541298 7067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:39.542460 7067 hybrid_clock.cc:648] HybridClock initialized: now 1782028059542443 us; error 27 us; skew 500 ppm
Jun 21 07:47:39 dist-test-slave-lzx3 krb5kdc[6826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028059, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.192.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:39.544718 7067 init.cc:377] Logged in from keytab as oryx/127.0.192.194@KRBTEST.COM (short username oryx)
I20260621 07:47:39.545469 7067 webserver.cc:460] Webserver started at http://127.0.192.194:43591/ using document root <none> and password file <none>
I20260621 07:47:39.545926 7067 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:39.546002 7067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:39.546177 7067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:39.547837 7067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-1/data/instance:
uuid: "65c308b30fa54aaba7bd5203d9b258a1"
format_stamp: "Formatted at 2026-06-21 07:47:39 on dist-test-slave-lzx3"
server_key: "1b47ac7a137f994bcfcff826d628b6fb"
server_key_iv: "35de321f802bd747ef0abb44f73025af"
server_key_version: "encryptionkey@0"
I20260621 07:47:39.548357 7067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-1/wal/instance:
uuid: "65c308b30fa54aaba7bd5203d9b258a1"
format_stamp: "Formatted at 2026-06-21 07:47:39 on dist-test-slave-lzx3"
server_key: "1b47ac7a137f994bcfcff826d628b6fb"
server_key_iv: "35de321f802bd747ef0abb44f73025af"
server_key_version: "encryptionkey@0"
I20260621 07:47:39.551277 7067 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260621 07:47:39.553210 7082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:39.554196 7067 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:39.554314 7067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-1/wal
uuid: "65c308b30fa54aaba7bd5203d9b258a1"
format_stamp: "Formatted at 2026-06-21 07:47:39 on dist-test-slave-lzx3"
server_key: "1b47ac7a137f994bcfcff826d628b6fb"
server_key_iv: "35de321f802bd747ef0abb44f73025af"
server_key_version: "encryptionkey@0"
I20260621 07:47:39.554406 7067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:39.572466 7067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:39.574684 7067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:39.574858 7067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:39.575340 7067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:47:39.576036 7067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:47:39.576097 7067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:39.576148 7067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:47:39.576186 7067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:39.584256 7067 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.194:40815
I20260621 07:47:39.584276 7195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.194:40815 every 8 connection(s)
I20260621 07:47:39.585018 7067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
Jun 21 07:47:39 dist-test-slave-lzx3 krb5kdc[6826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028059, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.192.194@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM
I20260621 07:47:39.593050 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 7067
I20260621 07:47:39.593156 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-1/wal/instance
I20260621 07:47:39.593470 771 external_mini_cluster.cc:1428] Setting key 316d86503955b361e5e5d20cfc029cd1
I20260621 07:47:39.597311 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:39.586614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:36319 (local address 127.0.192.254:40701)
0621 07:47:39.586729 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:39.586733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:39.587850 (+ 1117us) server_negotiation.cc:407] Connection header received
0621 07:47:39.588621 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:39.588624 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:39.588671 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:39.588734 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:39.589623 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:39.590088 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:39.591069 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:39.591279 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:39.591368 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:39.593673 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:39.593676 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:39.593678 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:39.593699 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:39.595088 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:39.595693 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:39.595696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:39.595697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:39.595765 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:39.596141 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:39.596143 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:39.596144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:39.596287 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:39.596354 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:39.596972 (+ 618us) server_negotiation.cc:299] Negotiation successful
0621 07:47:39.597091 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260621 07:47:39.597965 7198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:39.586939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:40701 (local address 127.0.192.194:36319)
0621 07:47:39.587710 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:39.587747 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:39.588411 (+ 664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:39.588839 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:39.588845 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:39.589134 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:39.589508 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:39.589519 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:39.590180 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:39.590182 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:39.590966 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:39.590974 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:39.591400 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:39.591402 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:39.591523 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:39.591952 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:39.591966 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:39.593514 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:39.595196 (+ 1682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:39.595202 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:39.595215 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:39.595593 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:39.595861 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:39.595867 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:39.595872 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:39.596051 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:39.596380 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:39.596388 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:39.596707 (+ 319us) client_negotiation.cc:769] Sending connection context
0621 07:47:39.596906 (+ 199us) client_negotiation.cc:240] Negotiation successful
0621 07:47:39.597121 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":500,"threads_started":1}
I20260621 07:47:39.599108 7196 heartbeater.cc:347] Connected to a master server at 127.0.192.254:40701
I20260621 07:47:39.599339 7196 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:39.599845 7196 heartbeater.cc:511] Master 127.0.192.254:40701 requested a full tablet report, sending...
I20260621 07:47:39.600870 6871 ts_manager.cc:194] Registered new tserver with Master: 65c308b30fa54aaba7bd5203d9b258a1 (127.0.192.194:40815)
I20260621 07:47:39.601373 6871 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.192.194@KRBTEST.COM'} at 127.0.192.194:36319
WARNING: no policy specified for oryx/127.0.192.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.192.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.192.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/oryx_127.0.192.195.keytab.
Entry for principal oryx/127.0.192.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/oryx_127.0.192.195.keytab.
I20260621 07:47:39.618678 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.195:0
--local_ip_for_outbound_sockets=127.0.192.195
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:40701
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/oryx_127.0.192.195.keytab
--principal=oryx/127.0.192.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:39257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:39.722164 7201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:39.722378 7201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:39.722442 7201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:39.726017 7201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:39.726061 7201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:39.726136 7201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.195
I20260621 07:47:39.730791 7201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:39257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/oryx_127.0.192.195.keytab
--principal=oryx/127.0.192.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:40701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.195
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:39.731628 7201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:39.732450 7201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:39.740190 7207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:39.740193 7206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:39.740371 7209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:39.740849 7201 server_base.cc:1034] running on GCE node
I20260621 07:47:39.741175 7201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:39.741659 7201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:39.742818 7201 hybrid_clock.cc:648] HybridClock initialized: now 1782028059742802 us; error 35 us; skew 500 ppm
Jun 21 07:47:39 dist-test-slave-lzx3 krb5kdc[6826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028059, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.192.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:39.745064 7201 init.cc:377] Logged in from keytab as oryx/127.0.192.195@KRBTEST.COM (short username oryx)
I20260621 07:47:39.745720 7201 webserver.cc:460] Webserver started at http://127.0.192.195:44405/ using document root <none> and password file <none>
I20260621 07:47:39.746152 7201 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:39.746227 7201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:39.746412 7201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:39.748085 7201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-2/data/instance:
uuid: "eecd298bb09548e0b4f4edffda7cbe91"
format_stamp: "Formatted at 2026-06-21 07:47:39 on dist-test-slave-lzx3"
server_key: "6f0c5d196a0a1d48dbab76dd241e8658"
server_key_iv: "e9ed6255ccf78561234211f9556b0205"
server_key_version: "encryptionkey@0"
I20260621 07:47:39.748661 7201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-2/wal/instance:
uuid: "eecd298bb09548e0b4f4edffda7cbe91"
format_stamp: "Formatted at 2026-06-21 07:47:39 on dist-test-slave-lzx3"
server_key: "6f0c5d196a0a1d48dbab76dd241e8658"
server_key_iv: "e9ed6255ccf78561234211f9556b0205"
server_key_version: "encryptionkey@0"
I20260621 07:47:39.751730 7201 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260621 07:47:39.753688 7216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:39.754673 7201 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 07:47:39.754794 7201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-2/wal
uuid: "eecd298bb09548e0b4f4edffda7cbe91"
format_stamp: "Formatted at 2026-06-21 07:47:39 on dist-test-slave-lzx3"
server_key: "6f0c5d196a0a1d48dbab76dd241e8658"
server_key_iv: "e9ed6255ccf78561234211f9556b0205"
server_key_version: "encryptionkey@0"
I20260621 07:47:39.754887 7201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:39.770008 7201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:39.772270 7201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:39.772449 7201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:39.772943 7201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:47:39.773643 7201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:47:39.773702 7201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:39.773754 7201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:47:39.773793 7201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:39.781764 7201 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.195:43223
I20260621 07:47:39.781802 7329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.195:43223 every 8 connection(s)
I20260621 07:47:39.782467 7201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
I20260621 07:47:39.782970 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 7201
I20260621 07:47:39.783100 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-2/wal/instance
I20260621 07:47:39.783408 771 external_mini_cluster.cc:1428] Setting key 4526773340203762f1815cf70e34ac72
Jun 21 07:47:39 dist-test-slave-lzx3 krb5kdc[6826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028059, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.192.195@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM
I20260621 07:47:39.793236 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:39.784073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:47783 (local address 127.0.192.254:40701)
0621 07:47:39.784214 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:39.784218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:39.784939 (+ 721us) server_negotiation.cc:407] Connection header received
0621 07:47:39.786095 (+ 1156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:39.786097 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:39.786139 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:39.786205 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:39.787040 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:39.787563 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:39.788611 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:39.788809 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:39.788898 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:39.790751 (+ 1853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:39.790755 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:39.790757 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:39.790777 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:39.791791 (+ 1014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:39.792164 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:39.792166 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:39.792167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:39.792229 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:39.792468 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:39.792470 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:39.792471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:39.792650 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:39.792717 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:39.793060 (+ 343us) server_negotiation.cc:299] Negotiation successful
0621 07:47:39.793139 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260621 07:47:39.793799 7332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:39.784381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:40701 (local address 127.0.192.195:47783)
0621 07:47:39.784806 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:39.784847 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:39.785936 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:39.786304 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:39.786309 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:39.786581 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:39.786936 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:39.786946 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:39.787665 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:39.787667 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:39.788493 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:39.788500 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:39.788938 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:39.788940 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:39.789055 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:39.789444 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:39.789456 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:39.790624 (+ 1168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:39.791877 (+ 1253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:39.791881 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:39.791889 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:39.792091 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:39.792293 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:39.792295 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:39.792296 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:39.792405 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:39.792723 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:39.792726 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:39.792898 (+ 172us) client_negotiation.cc:769] Sending connection context
0621 07:47:39.793022 (+ 124us) client_negotiation.cc:240] Negotiation successful
0621 07:47:39.793201 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":141,"threads_started":1}
I20260621 07:47:39.794659 7330 heartbeater.cc:347] Connected to a master server at 127.0.192.254:40701
I20260621 07:47:39.794827 7330 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:39.795163 7330 heartbeater.cc:511] Master 127.0.192.254:40701 requested a full tablet report, sending...
I20260621 07:47:39.796021 6871 ts_manager.cc:194] Registered new tserver with Master: eecd298bb09548e0b4f4edffda7cbe91 (127.0.192.195:43223)
I20260621 07:47:39.796613 6871 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.192.195@KRBTEST.COM'} at 127.0.192.195:47783
I20260621 07:47:39.796854 771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.192.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.192.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.192.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/oryx_127.0.192.253.keytab.
Entry for principal oryx/127.0.192.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/oryx_127.0.192.253.keytab.
I20260621 07:47:39.819005 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.253:43105
--webserver_interface=127.0.192.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/oryx_127.0.192.253.keytab
--principal=oryx/127.0.192.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:39257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:40701,127.0.192.253:43105
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:39.911087 7335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:39.911324 7335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:39.911398 7335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:39.915467 7335 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:47:39.915548 7335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:39.915591 7335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:39.915648 7335 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:47:39.915691 7335 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:47:39.920707 7335 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:39257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:40701,127.0.192.253:43105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/oryx_127.0.192.253.keytab
--principal=oryx/127.0.192.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.253:43105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.192.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
W20260621 07:47:39.920912 7335 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.192.254:40701,127.0.192.253:43105'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260621 07:47:39.921506 7335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:39.922693 7335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:39.930576 7343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:39.930603 7340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:39.930620 7341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:39.930913 7335 server_base.cc:1034] running on GCE node
I20260621 07:47:39.931416 7335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:39.932331 7335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:39.933485 7335 hybrid_clock.cc:648] HybridClock initialized: now 1782028059933464 us; error 34 us; skew 500 ppm
Jun 21 07:47:39 dist-test-slave-lzx3 krb5kdc[6826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028059, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.192.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:39.935979 7335 init.cc:377] Logged in from keytab as oryx/127.0.192.253@KRBTEST.COM (short username oryx)
I20260621 07:47:39.936826 7335 webserver.cc:460] Webserver started at http://127.0.192.253:40963/ using document root <none> and password file <none>
I20260621 07:47:39.937476 7335 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:39.937553 7335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:39.937837 7335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:39.939688 7335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-1/data/instance:
uuid: "2b7be1241e6d426d87e026a0d4b8292c"
format_stamp: "Formatted at 2026-06-21 07:47:39 on dist-test-slave-lzx3"
server_key: "baf35e2a48755b4c516545467ce1dec1"
server_key_iv: "b9e224042d0bdaa730a9ce55986c1e7f"
server_key_version: "encryptionkey@0"
I20260621 07:47:39.940306 7335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-1/wal/instance:
uuid: "2b7be1241e6d426d87e026a0d4b8292c"
format_stamp: "Formatted at 2026-06-21 07:47:39 on dist-test-slave-lzx3"
server_key: "baf35e2a48755b4c516545467ce1dec1"
server_key_iv: "b9e224042d0bdaa730a9ce55986c1e7f"
server_key_version: "encryptionkey@0"
I20260621 07:47:39.944111 7335 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:47:39.946252 7350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:39.947316 7335 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:39.947438 7335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-1/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-1/wal
uuid: "2b7be1241e6d426d87e026a0d4b8292c"
format_stamp: "Formatted at 2026-06-21 07:47:39 on dist-test-slave-lzx3"
server_key: "baf35e2a48755b4c516545467ce1dec1"
server_key_iv: "b9e224042d0bdaa730a9ce55986c1e7f"
server_key_version: "encryptionkey@0"
I20260621 07:47:39.947527 7335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:39.957897 7335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:39.960366 7335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:39.960544 7335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:39.966921 7335 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.253:43105
I20260621 07:47:39.967152 7402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.253:43105 every 8 connection(s)
I20260621 07:47:39.967777 7335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-1/data/info.pb
I20260621 07:47:39.970146 7403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:39.973263 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 7335
I20260621 07:47:39.973414 7403 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 07:47:39 dist-test-slave-lzx3 krb5kdc[6826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028059, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.192.253@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM
Jun 21 07:47:39 dist-test-slave-lzx3 krb5kdc[6826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.192.253@KRBTEST.COM
I20260621 07:47:39.986248 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:39.975151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34144 (local address 127.0.192.254:40701)
0621 07:47:39.975272 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:39.975277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:39.975547 (+ 270us) server_negotiation.cc:407] Connection header received
0621 07:47:39.975904 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:39.975906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:39.975956 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:39.976026 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:39.977218 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:39.977721 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:39.978743 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:39.978984 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:39.979060 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:39.982025 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:39.982049 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:39.982053 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:39.982084 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:39.983811 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:39.984501 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:39.984506 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:39.984508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:39.984626 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:39.984985 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:39.984989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:39.984991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:39.985296 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:39.985410 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:39.985934 (+ 524us) server_negotiation.cc:299] Negotiation successful
0621 07:47:39.986060 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260621 07:47:39.987057 7407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:39.975023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:40701 (local address 127.0.0.1:34144)
0621 07:47:39.975396 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:39.975441 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:39.975718 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:39.976132 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:39.976144 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:39.976592 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:39.977076 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:39.977089 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:39.977845 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:39.977848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:39.978615 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:39.978622 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:39.979149 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:39.979154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:39.979380 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:39.980201 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:39.980217 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:39.981893 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:39.984065 (+ 2172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:39.984074 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:39.984083 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:39.984390 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:39.984747 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:39.984750 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:39.984752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:39.984886 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:39.985425 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:39.985433 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:39.985685 (+ 252us) client_negotiation.cc:769] Sending connection context
0621 07:47:39.985842 (+ 157us) client_negotiation.cc:240] Negotiation successful
0621 07:47:39.986127 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":50,"thread_start_us":102,"threads_started":1}
I20260621 07:47:39.987057 7406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:39.974520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40176 (local address 127.0.192.253:43105)
0621 07:47:39.974881 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:39.974893 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:39.974920 (+ 27us) server_negotiation.cc:407] Connection header received
0621 07:47:39.975718 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:39.975737 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:39.975948 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:39.976217 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:39.976965 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:39.977945 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:39.978778 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:39.979335 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:39.979480 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:39.982622 (+ 3142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:39.982631 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:39.982645 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:39.982673 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:39.984548 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:39.985086 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:39.985090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:39.985091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:39.985155 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:39.985451 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:39.985454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:39.985456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:39.985693 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:39.985908 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:39.986139 (+ 231us) server_negotiation.cc:299] Negotiation successful
0621 07:47:39.986242 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":90,"server-negotiator.queue_time_us":262,"thread_start_us":137,"threads_started":1}
I20260621 07:47:39.988481 7403 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.192.253" port: 43105 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 07:47:39 dist-test-slave-lzx3 krb5kdc[6826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028059, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.192.253@KRBTEST.COM for oryx/127.0.192.253@KRBTEST.COM
I20260621 07:47:39.995402 7407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:39.988926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.253:43105 (local address 127.0.0.1:40180)
0621 07:47:39.989033 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:39.989050 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:39.989119 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:39.989328 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:39.989332 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:39.989485 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:39.989569 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:39.989574 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:39.990195 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:39.990197 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:39.990687 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:39.990695 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:39.991103 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:39.991106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:39.991173 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:39.991624 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:39.991640 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:39.992793 (+ 1153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:39.994215 (+ 1422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:39.994218 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:39.994219 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:39.994416 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:39.994656 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:39.994658 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:39.994660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:39.994732 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:39.995022 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:39.995024 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:39.995091 (+ 67us) client_negotiation.cc:769] Sending connection context
0621 07:47:39.995175 (+ 84us) client_negotiation.cc:240] Negotiation successful
0621 07:47:39.995271 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260621 07:47:39.995432 7406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:39.989017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40180 (local address 127.0.192.253:43105)
0621 07:47:39.989119 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:39.989123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:39.989136 (+ 13us) server_negotiation.cc:407] Connection header received
0621 07:47:39.989194 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:39.989195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:39.989224 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:39.989284 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:39.989668 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:39.990107 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:39.990794 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:39.991010 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:39.991088 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:39.992910 (+ 1822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:39.992913 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:39.992915 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:39.992934 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:39.994115 (+ 1181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:39.994505 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:39.994508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:39.994509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:39.994573 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:39.994806 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:39.994808 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:39.994809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:39.994945 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:39.995015 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:39.995222 (+ 207us) server_negotiation.cc:299] Negotiation successful
0621 07:47:39.995319 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":38}
I20260621 07:47:39.998095 7403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c: Bootstrap starting.
I20260621 07:47:40.000039 7403 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:40.000774 7403 log.cc:824] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c: Log is configured to *not* fsync() on all Append() calls
I20260621 07:47:40.002337 7403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c: No bootstrap required, opened a new log
I20260621 07:47:40.005002 7403 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 43105 } }
I20260621 07:47:40.005157 7403 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:40.005232 7403 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b7be1241e6d426d87e026a0d4b8292c, State: Initialized, Role: FOLLOWER
I20260621 07:47:40.005620 7403 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 43105 } }
I20260621 07:47:40.006011 7409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 43105 } } }
I20260621 07:47:40.006109 7409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:47:40.006608 7403 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [sys.catalog]: configured and running, proceeding with master startup.
W20260621 07:47:40.010833 7420 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:47:40.010910 7420 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:47:40.012827 7335 master_runner.cc:428] Detected that this master 2b7be1241e6d426d87e026a0d4b8292c is joining an existing cluster
I20260621 07:47:40.012893 7335 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.192.253:43105
I20260621 07:47:40.022989 7406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.014951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40190 (local address 127.0.192.253:43105)
0621 07:47:40.015045 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:40.015049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:40.015615 (+ 566us) server_negotiation.cc:407] Connection header received
0621 07:47:40.015771 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:40.015774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:40.015813 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:40.015897 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:40.016315 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.016839 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.018177 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.018369 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.018424 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.019699 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:40.019703 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:40.019707 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:40.019731 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:40.021360 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.021771 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.021777 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.021779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.021849 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.022140 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.022143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.022145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.022335 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:40.022432 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:40.022703 (+ 271us) server_negotiation.cc:299] Negotiation successful
0621 07:47:40.022833 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":39}
I20260621 07:47:40.023002 7427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.015216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.253:43105 (local address 127.0.0.1:40190)
0621 07:47:40.015527 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:40.015537 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:40.015672 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:40.015922 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:40.015925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:40.016125 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:40.016211 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.016216 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.016946 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.016952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.018054 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.018064 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.018524 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.018526 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.018615 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.019081 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:40.019092 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:40.019564 (+ 472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:40.021485 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:40.021490 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:40.021492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:40.021679 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:40.021943 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:40.021946 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:40.021948 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:40.022038 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:40.022441 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:40.022446 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:40.022540 (+ 94us) client_negotiation.cc:769] Sending connection context
0621 07:47:40.022652 (+ 112us) client_negotiation.cc:240] Negotiation successful
0621 07:47:40.022777 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":111,"spinlock_wait_cycles":1792,"thread_start_us":85,"threads_started":1}
I20260621 07:47:40.023501 7426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.014877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:40701 (local address 127.0.0.1:34158)
0621 07:47:40.015244 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:40.015258 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:40.015356 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:40.015612 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:40.015615 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:40.015819 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:40.015964 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.015974 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.016986 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.016990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.018054 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.018064 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.018628 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.018631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.018731 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.019265 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:40.019280 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:40.019663 (+ 383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:40.022008 (+ 2345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:40.022010 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:40.022012 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:40.022223 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:40.022500 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:40.022502 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:40.022504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:40.022588 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:40.023031 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:40.023033 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:40.023109 (+ 76us) client_negotiation.cc:769] Sending connection context
0621 07:47:40.023207 (+ 98us) client_negotiation.cc:240] Negotiation successful
0621 07:47:40.023330 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":81,"spinlock_wait_cycles":896,"thread_start_us":102,"threads_started":1}
I20260621 07:47:40.023651 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.014959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34158 (local address 127.0.192.254:40701)
0621 07:47:40.015134 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:40.015138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:40.015343 (+ 205us) server_negotiation.cc:407] Connection header received
0621 07:47:40.015445 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:40.015448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:40.015492 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:40.015575 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:40.016087 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.016889 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.018185 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.018517 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.018623 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.019794 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:40.019798 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:40.019801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:40.019825 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:40.021911 (+ 2086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.022315 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.022319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.022321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.022424 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.022697 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.022700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.022702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.022896 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:40.022961 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:40.023285 (+ 324us) server_negotiation.cc:299] Negotiation successful
0621 07:47:40.023414 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
Jun 21 07:47:40 dist-test-slave-lzx3 krb5kdc[6826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028059, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.192.254@KRBTEST.COM for oryx/127.0.192.253@KRBTEST.COM
I20260621 07:47:40.035804 7406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.027260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40202 (local address 127.0.192.253:43105)
0621 07:47:40.027353 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:40.027357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:40.027981 (+ 624us) server_negotiation.cc:407] Connection header received
0621 07:47:40.028149 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:40.028151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:40.028188 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:40.028245 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:40.028918 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.029335 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.030482 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.030667 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.030734 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.032870 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:40.032873 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:40.032874 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:40.032893 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:40.034188 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.034660 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.034665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.034671 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.034736 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.035005 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.035010 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.035012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.035157 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:40.035255 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:40.035570 (+ 315us) server_negotiation.cc:299] Negotiation successful
0621 07:47:40.035668 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260621 07:47:40.035830 7428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.027218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.253:43105 (local address 127.0.0.1:40202)
0621 07:47:40.027855 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:40.027912 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:40.028030 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:40.028297 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:40.028302 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:40.028697 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:40.028803 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.028810 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.029427 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.029430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.030363 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.030372 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.030766 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.030770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.030878 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.031322 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:40.031335 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:40.032737 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:40.034301 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:40.034311 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:40.034315 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:40.034539 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:40.034833 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:40.034836 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:40.034837 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:40.034919 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:40.035270 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:40.035277 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:40.035378 (+ 101us) client_negotiation.cc:769] Sending connection context
0621 07:47:40.035510 (+ 132us) client_negotiation.cc:240] Negotiation successful
0621 07:47:40.035671 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":73,"threads_started":1}
I20260621 07:47:40.036885 6871 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.192.253:43105: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c"
member_type: NON_VOTER
last_known_addr {
host: "127.0.192.253"
port: 43105
}
attrs {
promote: true
}
}
dest_uuid: "c8072ff3594143ca8e36eb920eae693c"
cas_config_opid_index: -1
I20260621 07:47:40.037410 6871 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: NON_VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: true } }
I20260621 07:47:40.038798 7430 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 2b7be1241e6d426d87e026a0d4b8292c (127.0.192.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: NON_VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: true } } }
I20260621 07:47:40.039379 7377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:40.040482 7430 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.192.253:43105
I20260621 07:47:40.040594 7377 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [term 1 FOLLOWER]: Refusing update from remote peer c8072ff3594143ca8e36eb920eae693c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260621 07:47:40.040666 7429 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c8072ff3594143ca8e36eb920eae693c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: NON_VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: true } } }
I20260621 07:47:40.040759 7429 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:40.041126 7335 master.cc:537] Master@127.0.192.253:43105 shutting down...
I20260621 07:47:40.043016 7429 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: NON_VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260621 07:47:40.044829 6855 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260621 07:47:40.045738 7335 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [term 1 FOLLOWER]: Raft consensus shutting down.
W20260621 07:47:40.045786 6855 consensus_peers.cc:487] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c -> Peer 2b7be1241e6d426d87e026a0d4b8292c (127.0.192.253:43105): Couldn't send request to peer 2b7be1241e6d426d87e026a0d4b8292c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260621 07:47:40.045994 7335 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:47:40.046108 7335 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c: stopping tablet replica
I20260621 07:47:40.050271 7335 master.cc:552] Master@127.0.192.253:43105 shutdown complete.
I20260621 07:47:40.050318 7335 master_runner.cc:305] Clearing existing system tablet
I20260621 07:47:40.050518 7335 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:40.050570 7335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
WARNING: no policy specified for oryx/127.0.192.252@KRBTEST.COM; defaulting to no policy
I20260621 07:47:40.051721 7335 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.192.252@KRBTEST.COM" created.
I20260621 07:47:40.053040 7435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:40.053251 7335 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:40.053368 7335 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 07:47:40.053401 7335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-1/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-1/wal
uuid: "2b7be1241e6d426d87e026a0d4b8292c"
format_stamp: "Formatted at 2026-06-21 07:47:39 on dist-test-slave-lzx3"
server_key: "baf35e2a48755b4c516545467ce1dec1"
server_key_iv: "b9e224042d0bdaa730a9ce55986c1e7f"
server_key_version: "encryptionkey@0"
I20260621 07:47:40.055168 7335 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:47:40.057646 7335 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260621 07:47:40.057724 7335 log.cc:1196] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260621 07:47:40.057955 7335 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c: Deleting consensus metadata
I20260621 07:47:40.058099 7335 master_runner.cc:315] Copying system tablet from 127.0.192.254:40701
I20260621 07:47:40.059173 7335 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.192.254:40701
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.192.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/oryx_127.0.192.252.keytab.
Entry for principal oryx/127.0.192.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/oryx_127.0.192.252.keytab.
I20260621 07:47:40.066685 7442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.059646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:40701 (local address 127.0.0.1:34172)
0621 07:47:40.059918 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:40.059932 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:40.060042 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:40.060390 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:40.060393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:40.060733 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:40.060820 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.060827 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.061609 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.061612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.062449 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.062459 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.062930 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.062934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.063008 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.063396 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:40.063409 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:40.063752 (+ 343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:40.065367 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:40.065371 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:40.065372 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:40.065594 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:40.065827 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:40.065830 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:40.065831 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:40.065912 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:40.066233 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:40.066235 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:40.066314 (+ 79us) client_negotiation.cc:769] Sending connection context
0621 07:47:40.066417 (+ 103us) client_negotiation.cc:240] Negotiation successful
0621 07:47:40.066523 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":82,"threads_started":1}
I20260621 07:47:40.067061 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.059673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34172 (local address 127.0.192.254:40701)
0621 07:47:40.059766 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:40.059770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:40.060023 (+ 253us) server_negotiation.cc:407] Connection header received
0621 07:47:40.060127 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:40.060130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:40.060189 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:40.060288 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:40.061034 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.061505 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.062568 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.062831 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.062907 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.063889 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:40.063892 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:40.063894 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:40.063912 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:40.065281 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.065679 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.065681 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.065682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.065746 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.065997 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.066000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.066001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.066151 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:40.066220 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:40.066510 (+ 290us) server_negotiation.cc:299] Negotiation successful
0621 07:47:40.066704 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260621 07:47:40.067373 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.252:36421
--webserver_interface=127.0.192.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/oryx_127.0.192.252.keytab
--principal=oryx/127.0.192.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:39257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:40701,127.0.192.253:43105,127.0.192.252:36421
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:47:40.067447 6891 tablet_copy_service.cc:140] P c8072ff3594143ca8e36eb920eae693c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2b7be1241e6d426d87e026a0d4b8292c ({username='oryx', principal='oryx/127.0.192.253@KRBTEST.COM'} at 127.0.0.1:34172)
I20260621 07:47:40.067621 6891 tablet_copy_service.cc:161] P c8072ff3594143ca8e36eb920eae693c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2b7be1241e6d426d87e026a0d4b8292c at {username='oryx', principal='oryx/127.0.192.253@KRBTEST.COM'} at 127.0.0.1:34172: session id = 2b7be1241e6d426d87e026a0d4b8292c-00000000000000000000000000000000
I20260621 07:47:40.069123 6891 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c: Tablet Copy: opened 0 blocks and 1 log segments
I20260621 07:47:40.070559 7335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260621 07:47:40.072995 7335 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c: tablet copy: Starting download of 0 data blocks...
I20260621 07:47:40.073164 7335 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c: tablet copy: Starting download of 1 WAL segments...
I20260621 07:47:40.074783 7335 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260621 07:47:40.076339 6891 tablet_copy_service.cc:342] P c8072ff3594143ca8e36eb920eae693c: Request end of tablet copy session 2b7be1241e6d426d87e026a0d4b8292c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.192.253@KRBTEST.COM'} at 127.0.0.1:34172
I20260621 07:47:40.076453 6891 tablet_copy_service.cc:433] P c8072ff3594143ca8e36eb920eae693c: ending tablet copy session 2b7be1241e6d426d87e026a0d4b8292c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2b7be1241e6d426d87e026a0d4b8292c
I20260621 07:47:40.082928 7335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:40.084180 7444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:40.085281 7445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:40.085459 7447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:40.091650 7335 server_base.cc:1034] running on GCE node
I20260621 07:47:40.091801 7335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:40.092023 7335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:40.093158 7335 hybrid_clock.cc:648] HybridClock initialized: now 1782028060093138 us; error 32 us; skew 500 ppm
Jun 21 07:47:40 dist-test-slave-lzx3 krb5kdc[6826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028060, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.192.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:40.094458 7335 init.cc:377] Logged in from keytab as oryx/127.0.192.253@KRBTEST.COM (short username oryx)
I20260621 07:47:40.094872 7335 webserver.cc:460] Webserver started at http://127.0.192.253:40821/ using document root <none> and password file <none>
I20260621 07:47:40.095041 7335 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:40.095081 7335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:40.096181 7335 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 07:47:40.097004 7454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:40.097254 7335 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260621 07:47:40.097323 7335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-1/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-1/wal
uuid: "2b7be1241e6d426d87e026a0d4b8292c"
format_stamp: "Formatted at 2026-06-21 07:47:39 on dist-test-slave-lzx3"
server_key: "baf35e2a48755b4c516545467ce1dec1"
server_key_iv: "b9e224042d0bdaa730a9ce55986c1e7f"
server_key_version: "encryptionkey@0"
I20260621 07:47:40.097396 7335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:40.129264 7335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:40.130978 7335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:40.135536 7335 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.253:43105
I20260621 07:47:40.135566 7506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.253:43105 every 8 connection(s)
I20260621 07:47:40.136088 7335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-1/data/info.pb
I20260621 07:47:40.137633 7507 sys_catalog.cc:263] Verifying existing consensus state
I20260621 07:47:40.138574 7507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c: Bootstrap starting.
I20260621 07:47:40.148175 7507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 07:47:40.148497 7507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c: Bootstrap complete.
I20260621 07:47:40.149489 7507 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: NON_VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: true } }
I20260621 07:47:40.149622 7507 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2b7be1241e6d426d87e026a0d4b8292c, State: Initialized, Role: LEARNER
I20260621 07:47:40.149722 7507 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: NON_VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: true } }
I20260621 07:47:40.150002 7510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: NON_VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: true } } }
I20260621 07:47:40.150123 7507 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:47:40.150190 7510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [sys.catalog]: This master's current role is: LEARNER
I20260621 07:47:40.153214 7521 catalog_manager.cc:1259] Loaded cluster ID: 2feba99244ef42a5b71acfd2c5120240
I20260621 07:47:40.153322 7521 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c: loading cluster ID for follower catalog manager: success
I20260621 07:47:40.155470 7521 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c: acquiring CA information for follower catalog manager: success
I20260621 07:47:40.156852 7521 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260621 07:47:40.178346 7443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:40.178604 7443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:40.178695 7443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:40.182451 7443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:47:40.182505 7443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:40.182544 7443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:40.182574 7443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:47:40.182603 7443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:47:40.187147 7443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:39257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:40701,127.0.192.253:43105,127.0.192.252:36421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/oryx_127.0.192.252.keytab
--principal=oryx/127.0.192.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.252:36421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.192.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:40.188014 7443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:40.188881 7443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:40.195637 7527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:40.195677 7526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:40.195713 7529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:40.195783 7443 server_base.cc:1034] running on GCE node
I20260621 07:47:40.196333 7443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:40.197265 7443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:40.198458 7443 hybrid_clock.cc:648] HybridClock initialized: now 1782028060198434 us; error 40 us; skew 500 ppm
Jun 21 07:47:40 dist-test-slave-lzx3 krb5kdc[6826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028060, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.192.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:40.200846 7443 init.cc:377] Logged in from keytab as oryx/127.0.192.252@KRBTEST.COM (short username oryx)
I20260621 07:47:40.201596 7443 webserver.cc:460] Webserver started at http://127.0.192.252:37275/ using document root <none> and password file <none>
I20260621 07:47:40.202085 7443 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:40.202159 7443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:40.202343 7443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:40.203992 7443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-2/data/instance:
uuid: "301f9ea85622455aaa68534f71280c4d"
format_stamp: "Formatted at 2026-06-21 07:47:40 on dist-test-slave-lzx3"
server_key: "ee1dd847bd91ebeabe56f8426bf26cb0"
server_key_iv: "77fa30fd51d5e80ef56b405f2b3c24a5"
server_key_version: "encryptionkey@0"
I20260621 07:47:40.204463 7443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-2/wal/instance:
uuid: "301f9ea85622455aaa68534f71280c4d"
format_stamp: "Formatted at 2026-06-21 07:47:40 on dist-test-slave-lzx3"
server_key: "ee1dd847bd91ebeabe56f8426bf26cb0"
server_key_iv: "77fa30fd51d5e80ef56b405f2b3c24a5"
server_key_version: "encryptionkey@0"
I20260621 07:47:40.207302 7443 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260621 07:47:40.209391 7536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:40.210338 7443 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:40.210475 7443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-2/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-2/wal
uuid: "301f9ea85622455aaa68534f71280c4d"
format_stamp: "Formatted at 2026-06-21 07:47:40 on dist-test-slave-lzx3"
server_key: "ee1dd847bd91ebeabe56f8426bf26cb0"
server_key_iv: "77fa30fd51d5e80ef56b405f2b3c24a5"
server_key_version: "encryptionkey@0"
I20260621 07:47:40.210564 7443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:40.247812 7443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:40.250242 7443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:40.250403 7443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:40.256203 7443 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.252:36421
I20260621 07:47:40.256419 7588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.252:36421 every 8 connection(s)
I20260621 07:47:40.257007 7443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-2/data/info.pb
I20260621 07:47:40.259636 7589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:40.262737 7589 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:47:40.263293 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 7443
Jun 21 07:47:40 dist-test-slave-lzx3 krb5kdc[6826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028060, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.192.252@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM
Jun 21 07:47:40 dist-test-slave-lzx3 krb5kdc[6826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.192.252@KRBTEST.COM
I20260621 07:47:40.274896 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.264518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34186 (local address 127.0.192.254:40701)
0621 07:47:40.264646 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:40.264650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:40.265156 (+ 506us) server_negotiation.cc:407] Connection header received
0621 07:47:40.266324 (+ 1168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:40.266327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:40.266373 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:40.266444 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:40.267173 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.267700 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.268784 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.269030 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.269104 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.271433 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:40.271437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:40.271439 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:40.271463 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:40.272964 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.273470 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.273473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.273475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.273554 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.273897 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.273899 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.273901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.274120 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:40.274180 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:40.274607 (+ 427us) server_negotiation.cc:299] Negotiation successful
0621 07:47:40.274751 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260621 07:47:40.275616 7591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.264513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:40701 (local address 127.0.0.1:34186)
0621 07:47:40.264969 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:40.265012 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:40.266154 (+ 1142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:40.266498 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:40.266502 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:40.266836 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:40.267043 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.267057 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.267822 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.267825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.268686 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.268694 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.269138 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.269141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.269295 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.269905 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:40.269920 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:40.271267 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:40.273064 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:40.273071 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:40.273077 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:40.273382 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:40.273635 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:40.273640 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:40.273644 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:40.273794 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:40.274208 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:40.274214 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:40.274399 (+ 185us) client_negotiation.cc:769] Sending connection context
0621 07:47:40.274551 (+ 152us) client_negotiation.cc:240] Negotiation successful
0621 07:47:40.274847 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":40,"spinlock_wait_cycles":45184,"thread_start_us":140,"threads_started":1}
I20260621 07:47:40.275616 7592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.264512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53830 (local address 127.0.192.252:36421)
0621 07:47:40.265248 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:40.265264 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:40.265290 (+ 26us) server_negotiation.cc:407] Connection header received
0621 07:47:40.266164 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:40.266180 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:40.266324 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:40.266473 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:40.267058 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.268008 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.268903 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.269249 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.269334 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.271858 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:40.271866 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:40.271877 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:40.271899 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:40.273463 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.273905 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.273910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.273911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.273994 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.274239 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.274241 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.274242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.274431 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:40.274581 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:40.274822 (+ 241us) server_negotiation.cc:299] Negotiation successful
0621 07:47:40.274944 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":650,"thread_start_us":515,"threads_started":1}
I20260621 07:47:40.276733 7589 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.192.253" port: 43105 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 07:47:40 dist-test-slave-lzx3 krb5kdc[6826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028060, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.192.252@KRBTEST.COM for oryx/127.0.192.253@KRBTEST.COM
I20260621 07:47:40.284175 7591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.277102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.253:43105 (local address 127.0.0.1:40214)
0621 07:47:40.277343 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:40.277376 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:40.277444 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:40.277797 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:40.277800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:40.277959 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:40.278032 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.278037 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.278887 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.278890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.279350 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.279357 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.279767 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.279770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.279853 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.280198 (+ 345us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:40.280207 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:40.281317 (+ 1110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:40.282896 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:40.282899 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:40.282901 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:40.283069 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:40.283340 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:40.283344 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:40.283346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:40.283417 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:40.283749 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:40.283752 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:40.283823 (+ 71us) client_negotiation.cc:769] Sending connection context
0621 07:47:40.283940 (+ 117us) client_negotiation.cc:240] Negotiation successful
0621 07:47:40.284053 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173}
I20260621 07:47:40.284221 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.277216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40214 (local address 127.0.192.253:43105)
0621 07:47:40.277608 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:40.277611 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:40.277622 (+ 11us) server_negotiation.cc:407] Connection header received
0621 07:47:40.277655 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:40.277657 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:40.277691 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:40.277786 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:40.278149 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.278779 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.279460 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.279653 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.279767 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.281466 (+ 1699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:40.281468 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:40.281470 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:40.281489 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:40.282784 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.283154 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.283158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.283160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.283271 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.283507 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.283510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.283511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.283665 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:40.283757 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:40.283986 (+ 229us) server_negotiation.cc:299] Negotiation successful
0621 07:47:40.284085 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":89,"threads_started":1}
I20260621 07:47:40.285050 7589 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.192.252" port: 36421 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 07:47:40 dist-test-slave-lzx3 krb5kdc[6826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028060, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.192.252@KRBTEST.COM for oryx/127.0.192.252@KRBTEST.COM
I20260621 07:47:40.291018 7591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.285390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:36421 (local address 127.0.0.1:53832)
0621 07:47:40.285481 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:40.285508 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:40.285573 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:40.285761 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:40.285764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:40.285909 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:40.285960 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.285964 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.286535 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.286540 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.286942 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.286954 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.287350 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.287353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.287447 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.287790 (+ 343us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:40.287800 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:40.288851 (+ 1051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:40.290034 (+ 1183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:40.290036 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:40.290037 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:40.290195 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:40.290392 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:40.290393 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:40.290394 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:40.290453 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:40.290703 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:40.290704 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:40.290759 (+ 55us) client_negotiation.cc:769] Sending connection context
0621 07:47:40.290823 (+ 64us) client_negotiation.cc:240] Negotiation successful
0621 07:47:40.290904 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":4}
I20260621 07:47:40.291062 7592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.285475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53832 (local address 127.0.192.252:36421)
0621 07:47:40.285562 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:40.285565 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:40.285576 (+ 11us) server_negotiation.cc:407] Connection header received
0621 07:47:40.285647 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:40.285650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:40.285682 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:40.285753 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:40.286050 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.286438 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.287056 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.287242 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.287353 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.288957 (+ 1604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:40.288960 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:40.288962 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:40.288980 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:40.289944 (+ 964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.290265 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.290267 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.290268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.290327 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.290517 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.290519 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.290520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.290641 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:40.290704 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:40.290859 (+ 155us) server_negotiation.cc:299] Negotiation successful
0621 07:47:40.290946 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":34}
I20260621 07:47:40.293437 7589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d: Bootstrap starting.
I20260621 07:47:40.295292 7589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:40.295949 7589 log.cc:824] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d: Log is configured to *not* fsync() on all Append() calls
I20260621 07:47:40.297427 7589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d: No bootstrap required, opened a new log
I20260621 07:47:40.299934 7589 raft_consensus.cc:348] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 43105 } } peers { permanent_uuid: "301f9ea85622455aaa68534f71280c4d" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 36421 } }
I20260621 07:47:40.300112 7589 raft_consensus.cc:374] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:40.300180 7589 raft_consensus.cc:729] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 301f9ea85622455aaa68534f71280c4d, State: Initialized, Role: FOLLOWER
I20260621 07:47:40.300537 7589 consensus_queue.cc:260] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 43105 } } peers { permanent_uuid: "301f9ea85622455aaa68534f71280c4d" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 36421 } }
I20260621 07:47:40.300972 7595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 43105 } } peers { permanent_uuid: "301f9ea85622455aaa68534f71280c4d" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 36421 } } }
I20260621 07:47:40.301064 7595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:47:40.301515 7589 sys_catalog.cc:564] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [sys.catalog]: configured and running, proceeding with master startup.
W20260621 07:47:40.305163 7606 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:47:40.305228 7606 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:47:40.307690 7443 master_runner.cc:428] Detected that this master 301f9ea85622455aaa68534f71280c4d is joining an existing cluster
I20260621 07:47:40.307737 7443 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.192.252:36421
I20260621 07:47:40.317442 7614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.309995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:40701 (local address 127.0.0.1:34190)
0621 07:47:40.310996 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:40.311007 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:40.311111 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:40.311375 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:40.311380 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:40.311576 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:40.311732 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.311739 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.312638 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.312641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.313404 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.313414 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.313888 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.313890 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.313944 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.314391 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:40.314405 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:40.314680 (+ 275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:40.316297 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:40.316299 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:40.316300 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:40.316449 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:40.316715 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:40.316717 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:40.316718 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:40.316780 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:40.317110 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:40.317111 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:40.317177 (+ 66us) client_negotiation.cc:769] Sending connection context
0621 07:47:40.317243 (+ 66us) client_negotiation.cc:240] Negotiation successful
0621 07:47:40.317321 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":929,"mutex_wait_us":51,"thread_start_us":488,"threads_started":1}
I20260621 07:47:40.317569 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.309798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34190 (local address 127.0.192.254:40701)
0621 07:47:40.309889 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:40.309892 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:40.311063 (+ 1171us) server_negotiation.cc:407] Connection header received
0621 07:47:40.311193 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:40.311195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:40.311229 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:40.311280 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:40.311845 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.312513 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.313529 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.313743 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.313793 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.314819 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:40.314822 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:40.314825 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:40.314848 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:40.316199 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.316532 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.316535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.316537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.316634 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.316854 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.316857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.316859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.317030 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:40.317117 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:40.317309 (+ 192us) server_negotiation.cc:299] Negotiation successful
0621 07:47:40.317431 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Jun 21 07:47:40 dist-test-slave-lzx3 krb5kdc[6826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028059, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.192.254@KRBTEST.COM for oryx/127.0.192.252@KRBTEST.COM
I20260621 07:47:40.328426 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 6933
I20260621 07:47:40.335047 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.193:45293
--local_ip_for_outbound_sockets=127.0.192.193
--tserver_master_addrs=127.0.192.254:40701
--webserver_port=39811
--webserver_interface=127.0.192.193
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/oryx_127.0.192.193.keytab
--principal=oryx/127.0.192.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:39257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.192.254:40701,127.0.192.253:43105
--tserver_master_addrs=127.0.192.254:40701,127.0.192.253:43105,127.0.192.252:36421 with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:47:40.336582 7613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.309459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.253:43105 (local address 127.0.0.1:40222)
0621 07:47:40.309889 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:40.309901 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:40.309990 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:40.310210 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:40.310215 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:40.310418 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:40.310529 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.310535 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.311464 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.311468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.312247 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.312255 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.312821 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.312824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.312962 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.313370 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:40.313384 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:40.333158 (+ 19774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:40.334869 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:40.334874 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:40.334878 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:40.335173 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:40.335493 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:40.335495 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:40.335497 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:40.335583 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:40.336047 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:40.336050 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:40.336144 (+ 94us) client_negotiation.cc:769] Sending connection context
0621 07:47:40.336231 (+ 87us) client_negotiation.cc:240] Negotiation successful
0621 07:47:40.336356 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":21,"thread_start_us":101,"threads_started":1}
I20260621 07:47:40.337157 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.309504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40222 (local address 127.0.192.253:43105)
0621 07:47:40.309613 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:40.309617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:40.309952 (+ 335us) server_negotiation.cc:407] Connection header received
0621 07:47:40.310056 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:40.310061 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:40.310100 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:40.310628 (+ 528us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:40.310706 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.311356 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.312377 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.312706 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.313142 (+ 436us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.333280 (+ 20138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:40.333285 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:40.333289 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:40.333313 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:40.334750 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.335297 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.335302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.335305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.335386 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.335687 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.335691 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.335693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.335909 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:40.335992 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:40.336924 (+ 932us) server_negotiation.cc:299] Negotiation successful
0621 07:47:40.337031 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260621 07:47:40.337987 7612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.309471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:36421 (local address 127.0.0.1:53836)
0621 07:47:40.310177 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:40.310190 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:40.310255 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:40.310512 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:40.310514 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:40.310671 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:40.310760 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.310766 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.311618 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.311621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.312429 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.312438 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.313019 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.313022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.313370 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.313878 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:40.313891 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:40.332922 (+ 19031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:40.335815 (+ 2893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:40.335817 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:40.335819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:40.336050 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:40.336639 (+ 589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:40.336642 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:40.336643 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:40.336719 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:40.337472 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:40.337475 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:40.337568 (+ 93us) client_negotiation.cc:769] Sending connection context
0621 07:47:40.337653 (+ 85us) client_negotiation.cc:240] Negotiation successful
0621 07:47:40.337811 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":1,"thread_start_us":92,"threads_started":1}
I20260621 07:47:40.338339 7592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.309824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53836 (local address 127.0.192.252:36421)
0621 07:47:40.309900 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:40.309904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:40.310321 (+ 417us) server_negotiation.cc:407] Connection header received
0621 07:47:40.310345 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:40.310347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:40.310377 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:40.310431 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:40.310866 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.311517 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.312543 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.312913 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.314086 (+ 1173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.333140 (+ 19054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:40.333142 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:40.333145 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:40.333163 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:40.335721 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.336146 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.336148 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.336150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.336209 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.337021 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.337023 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.337025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.337178 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:40.337837 (+ 659us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:40.337901 (+ 64us) server_negotiation.cc:299] Negotiation successful
0621 07:47:40.337995 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":31}
I20260621 07:47:40.339833 7428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.320294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:36421 (local address 127.0.0.1:53844)
0621 07:47:40.320392 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:40.320406 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:40.320473 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:40.320780 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:40.320782 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:40.320930 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:40.321000 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.321005 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.321657 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.321660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.322128 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.322139 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.322616 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.322618 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.322673 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.323027 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:40.323036 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:40.324006 (+ 970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:40.338267 (+ 14261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:40.338270 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:40.338272 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:40.338531 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:40.338841 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:40.338847 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:40.338851 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:40.338952 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:40.339388 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:40.339391 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:40.339476 (+ 85us) client_negotiation.cc:769] Sending connection context
0621 07:47:40.339577 (+ 101us) client_negotiation.cc:240] Negotiation successful
0621 07:47:40.339697 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260621 07:47:40.340350 7615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.320346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53844 (local address 127.0.192.252:36421)
0621 07:47:40.320575 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:40.320585 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:40.320598 (+ 13us) server_negotiation.cc:407] Connection header received
0621 07:47:40.320647 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:40.320649 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:40.320681 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:40.320765 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:40.321097 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.321562 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.322238 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.322522 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.322609 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.324122 (+ 1513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:40.324125 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:40.324127 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:40.324148 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:40.338147 (+ 13999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.338661 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.338664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.338665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.338743 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.339080 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.339084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.339085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.339281 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:40.339381 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:40.339651 (+ 270us) server_negotiation.cc:299] Negotiation successful
0621 07:47:40.339768 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":74,"threads_started":1}
I20260621 07:47:40.341174 6871 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.192.252:36421: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "301f9ea85622455aaa68534f71280c4d"
member_type: NON_VOTER
last_known_addr {
host: "127.0.192.252"
port: 36421
}
attrs {
promote: true
}
}
dest_uuid: "c8072ff3594143ca8e36eb920eae693c"
cas_config_opid_index: 5
I20260621 07:47:40.341781 6871 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: NON_VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: true } } peers { permanent_uuid: "301f9ea85622455aaa68534f71280c4d" member_type: NON_VOTER last_known_addr { host: "127.0.192.252" port: 36421 } attrs { promote: true } }
I20260621 07:47:40.342859 7430 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 301f9ea85622455aaa68534f71280c4d (127.0.192.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: NON_VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: true } } peers { permanent_uuid: "301f9ea85622455aaa68534f71280c4d" member_type: NON_VOTER last_known_addr { host: "127.0.192.252" port: 36421 } attrs { promote: true } } }
I20260621 07:47:40.344378 7430 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.192.252:36421
I20260621 07:47:40.344542 7430 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c8072ff3594143ca8e36eb920eae693c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: NON_VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: true } } peers { permanent_uuid: "301f9ea85622455aaa68534f71280c4d" member_type: NON_VOTER last_known_addr { host: "127.0.192.252" port: 36421 } attrs { promote: true } } }
I20260621 07:47:40.344669 7430 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:40.345362 7563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:40.346626 7563 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [term 1 FOLLOWER]: Refusing update from remote peer c8072ff3594143ca8e36eb920eae693c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260621 07:47:40.347084 7430 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [LEADER]: Connected to new peer: Peer: permanent_uuid: "301f9ea85622455aaa68534f71280c4d" member_type: NON_VOTER last_known_addr { host: "127.0.192.252" port: 36421 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:47:40.347572 7428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.342983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.253:43105 (local address 127.0.0.1:40230)
0621 07:47:40.343416 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:40.343428 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:40.343494 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:40.343800 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:40.343803 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:40.343816 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:40.343876 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.343881 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.345038 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.345041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.346102 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.346112 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.347109 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.347112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.347255 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.347299 (+ 44us) client_negotiation.cc:769] Sending connection context
0621 07:47:40.347399 (+ 100us) client_negotiation.cc:240] Negotiation successful
0621 07:47:40.347460 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364}
I20260621 07:47:40.347942 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.343145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40230 (local address 127.0.192.253:43105)
0621 07:47:40.343237 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:40.343241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:40.343574 (+ 333us) server_negotiation.cc:407] Connection header received
0621 07:47:40.343608 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:40.343610 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:40.343651 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:40.343718 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:40.343977 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.344921 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.346258 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.346779 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.346878 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.347005 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:40.347779 (+ 774us) server_negotiation.cc:299] Negotiation successful
0621 07:47:40.347845 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260621 07:47:40.348631 7481 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [term 1 LEARNER]: Refusing update from remote peer c8072ff3594143ca8e36eb920eae693c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260621 07:47:40.349002 7430 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: NON_VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260621 07:47:40.350458 7563 raft_consensus.cc:858] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 43105 } } peers { permanent_uuid: "301f9ea85622455aaa68534f71280c4d" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 36421 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: NON_VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: NON_VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: true } } peers { permanent_uuid: "301f9ea85622455aaa68534f71280c4d" member_type: NON_VOTER last_known_addr { host: "127.0.192.252" port: 36421 } attrs { promote: true } }
W20260621 07:47:40.350539 7563 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 43105 } } peers { permanent_uuid: "301f9ea85622455aaa68534f71280c4d" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 36421 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: NON_VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: NON_VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: true } } peers { permanent_uuid: "301f9ea85622455aaa68534f71280c4d" member_type: NON_VOTER last_known_addr { host: "127.0.192.252" port: 36421 } attrs { promote: true } }
I20260621 07:47:40.350453 7481 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 301f9ea85622455aaa68534f71280c4d (127.0.192.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: NON_VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: true } } peers { permanent_uuid: "301f9ea85622455aaa68534f71280c4d" member_type: NON_VOTER last_known_addr { host: "127.0.192.252" port: 36421 } attrs { promote: true } } }
I20260621 07:47:40.351679 7563 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 2b7be1241e6d426d87e026a0d4b8292c (127.0.192.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: NON_VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: true } } }
I20260621 07:47:40.352344 7510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8072ff3594143ca8e36eb920eae693c. Latest consensus state: current_term: 1 leader_uuid: "c8072ff3594143ca8e36eb920eae693c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: NON_VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: true } } peers { permanent_uuid: "301f9ea85622455aaa68534f71280c4d" member_type: NON_VOTER last_known_addr { host: "127.0.192.252" port: 36421 } attrs { promote: true } } }
I20260621 07:47:40.352439 7510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [sys.catalog]: This master's current role is: LEARNER
I20260621 07:47:40.352515 7510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8072ff3594143ca8e36eb920eae693c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: NON_VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: true } } peers { permanent_uuid: "301f9ea85622455aaa68534f71280c4d" member_type: NON_VOTER last_known_addr { host: "127.0.192.252" port: 36421 } attrs { promote: true } } }
I20260621 07:47:40.352805 7510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [sys.catalog]: This master's current role is: LEARNER
I20260621 07:47:40.353104 7443 master.cc:537] Master@127.0.192.252:36421 shutting down...
I20260621 07:47:40.353111 7623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8072ff3594143ca8e36eb920eae693c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: NON_VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: true } } }
I20260621 07:47:40.353286 7623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260621 07:47:40.353361 7623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8072ff3594143ca8e36eb920eae693c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: NON_VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: true } } }
I20260621 07:47:40.353411 7623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260621 07:47:40.353394 7595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8072ff3594143ca8e36eb920eae693c. Latest consensus state: current_term: 1 leader_uuid: "c8072ff3594143ca8e36eb920eae693c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: NON_VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: true } } }
I20260621 07:47:40.353469 7595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260621 07:47:40.355257 7430 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8072ff3594143ca8e36eb920eae693c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: NON_VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: true } } peers { permanent_uuid: "301f9ea85622455aaa68534f71280c4d" member_type: NON_VOTER last_known_addr { host: "127.0.192.252" port: 36421 } attrs { promote: true } } }
I20260621 07:47:40.355346 7430 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [sys.catalog]: This master's current role is: LEADER
W20260621 07:47:40.356356 6857 consensus_peers.cc:487] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c -> Peer 301f9ea85622455aaa68534f71280c4d (127.0.192.252:36421): Couldn't send request to peer 301f9ea85622455aaa68534f71280c4d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260621 07:47:40.357422 7429 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c: attempting to promote NON_VOTER 2b7be1241e6d426d87e026a0d4b8292c to VOTER
I20260621 07:47:40.357767 7429 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: false } } peers { permanent_uuid: "301f9ea85622455aaa68534f71280c4d" member_type: NON_VOTER last_known_addr { host: "127.0.192.252" port: 36421 } attrs { promote: true } }
I20260621 07:47:40.359055 7481 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [term 1 LEARNER]: Refusing update from remote peer c8072ff3594143ca8e36eb920eae693c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260621 07:47:40.359232 6857 consensus_peers.cc:487] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c -> Peer 301f9ea85622455aaa68534f71280c4d (127.0.192.252:36421): Couldn't send request to peer 301f9ea85622455aaa68534f71280c4d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260621 07:47:40.359359 7429 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260621 07:47:40.359987 7443 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260621 07:47:40.360167 7443 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260621 07:47:40.360232 7443 tablet_replica.cc:330] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d: stopping tablet replica
I20260621 07:47:40.365607 7430 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2b7be1241e6d426d87e026a0d4b8292c (127.0.192.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: false } } peers { permanent_uuid: "301f9ea85622455aaa68534f71280c4d" member_type: NON_VOTER last_known_addr { host: "127.0.192.252" port: 36421 } attrs { promote: true } } }
I20260621 07:47:40.366703 7617 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c8072ff3594143ca8e36eb920eae693c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: false } } peers { permanent_uuid: "301f9ea85622455aaa68534f71280c4d" member_type: NON_VOTER last_known_addr { host: "127.0.192.252" port: 36421 } attrs { promote: true } } }
I20260621 07:47:40.366833 7617 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:40.366986 7430 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8072ff3594143ca8e36eb920eae693c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: false } } peers { permanent_uuid: "301f9ea85622455aaa68534f71280c4d" member_type: NON_VOTER last_known_addr { host: "127.0.192.252" port: 36421 } attrs { promote: true } } }
I20260621 07:47:40.367084 7430 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:40.367162 7481 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2b7be1241e6d426d87e026a0d4b8292c (127.0.192.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: false } } peers { permanent_uuid: "301f9ea85622455aaa68534f71280c4d" member_type: NON_VOTER last_known_addr { host: "127.0.192.252" port: 36421 } attrs { promote: true } } }
I20260621 07:47:40.368384 7622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8072ff3594143ca8e36eb920eae693c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: false } } peers { permanent_uuid: "301f9ea85622455aaa68534f71280c4d" member_type: NON_VOTER last_known_addr { host: "127.0.192.252" port: 36421 } attrs { promote: true } } }
I20260621 07:47:40.368487 7622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:47:40.378355 7443 master.cc:552] Master@127.0.192.252:36421 shutdown complete.
I20260621 07:47:40.378401 7443 master_runner.cc:305] Clearing existing system tablet
I20260621 07:47:40.378621 7443 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:40.378682 7443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:40.379725 7443 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 07:47:40.380429 7629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:40.380609 7443 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:40.380702 7443 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 07:47:40.380729 7443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-2/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-2/wal
uuid: "301f9ea85622455aaa68534f71280c4d"
format_stamp: "Formatted at 2026-06-21 07:47:40 on dist-test-slave-lzx3"
server_key: "ee1dd847bd91ebeabe56f8426bf26cb0"
server_key_iv: "77fa30fd51d5e80ef56b405f2b3c24a5"
server_key_version: "encryptionkey@0"
I20260621 07:47:40.382138 7443 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:47:40.383886 7443 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260621 07:47:40.383975 7443 log.cc:1196] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260621 07:47:40.384248 7443 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d: Deleting consensus metadata
I20260621 07:47:40.384392 7443 master_runner.cc:315] Copying system tablet from 127.0.192.254:40701
I20260621 07:47:40.385366 7443 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.192.254:40701
I20260621 07:47:40.393982 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.385866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34194 (local address 127.0.192.254:40701)
0621 07:47:40.385991 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:40.385997 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:40.386248 (+ 251us) server_negotiation.cc:407] Connection header received
0621 07:47:40.386376 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:40.386378 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:40.386419 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:40.386486 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:40.386988 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.387733 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.388877 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.389168 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.389254 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.390759 (+ 1505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:40.390762 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:40.390765 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:40.390787 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:40.392400 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.392879 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.392882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.392883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.392989 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.393253 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.393256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.393257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.393435 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:40.393533 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:40.393768 (+ 235us) server_negotiation.cc:299] Negotiation successful
0621 07:47:40.393859 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260621 07:47:40.393986 7635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.385766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:40701 (local address 127.0.0.1:34194)
0621 07:47:40.386143 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:40.386160 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:40.386273 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:40.386527 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:40.386530 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:40.386734 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:40.386851 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.386860 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.387849 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.387853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.388763 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.388772 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.389267 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.389270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.389339 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.390290 (+ 951us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:40.390302 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:40.390635 (+ 333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:40.392508 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:40.392510 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:40.392513 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:40.392780 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:40.393082 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:40.393084 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:40.393086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:40.393166 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:40.393536 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:40.393539 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:40.393618 (+ 79us) client_negotiation.cc:769] Sending connection context
0621 07:47:40.393712 (+ 94us) client_negotiation.cc:240] Negotiation successful
0621 07:47:40.393816 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":67,"threads_started":1}
I20260621 07:47:40.394866 6891 tablet_copy_service.cc:140] P c8072ff3594143ca8e36eb920eae693c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 301f9ea85622455aaa68534f71280c4d ({username='oryx', principal='oryx/127.0.192.252@KRBTEST.COM'} at 127.0.0.1:34194)
I20260621 07:47:40.394948 6891 tablet_copy_service.cc:161] P c8072ff3594143ca8e36eb920eae693c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 301f9ea85622455aaa68534f71280c4d at {username='oryx', principal='oryx/127.0.192.252@KRBTEST.COM'} at 127.0.0.1:34194: session id = 301f9ea85622455aaa68534f71280c4d-00000000000000000000000000000000
I20260621 07:47:40.395850 6891 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c: Tablet Copy: opened 0 blocks and 1 log segments
I20260621 07:47:40.396852 7443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260621 07:47:40.398658 7443 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d: tablet copy: Starting download of 0 data blocks...
I20260621 07:47:40.398799 7443 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d: tablet copy: Starting download of 1 WAL segments...
I20260621 07:47:40.399894 7443 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260621 07:47:40.401295 6891 tablet_copy_service.cc:342] P c8072ff3594143ca8e36eb920eae693c: Request end of tablet copy session 301f9ea85622455aaa68534f71280c4d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.192.252@KRBTEST.COM'} at 127.0.0.1:34194
I20260621 07:47:40.401360 6891 tablet_copy_service.cc:433] P c8072ff3594143ca8e36eb920eae693c: ending tablet copy session 301f9ea85622455aaa68534f71280c4d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 301f9ea85622455aaa68534f71280c4d
I20260621 07:47:40.406843 7443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:40.408330 7639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:40.408721 7636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:40.408804 7443 server_base.cc:1034] running on GCE node
W20260621 07:47:40.412494 7637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:40.412772 7443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:40.413041 7443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:40.414191 7443 hybrid_clock.cc:648] HybridClock initialized: now 1782028060414172 us; error 33 us; skew 500 ppm
Jun 21 07:47:40 dist-test-slave-lzx3 krb5kdc[6826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028060, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.192.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:40.415735 7443 init.cc:377] Logged in from keytab as oryx/127.0.192.252@KRBTEST.COM (short username oryx)
I20260621 07:47:40.416150 7443 webserver.cc:460] Webserver started at http://127.0.192.252:44561/ using document root <none> and password file <none>
I20260621 07:47:40.416302 7443 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:40.416339 7443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:40.417438 7443 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260621 07:47:40.418341 7646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:40.418598 7443 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 07:47:40.418691 7443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-2/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-2/wal
uuid: "301f9ea85622455aaa68534f71280c4d"
format_stamp: "Formatted at 2026-06-21 07:47:40 on dist-test-slave-lzx3"
server_key: "ee1dd847bd91ebeabe56f8426bf26cb0"
server_key_iv: "77fa30fd51d5e80ef56b405f2b3c24a5"
server_key_version: "encryptionkey@0"
I20260621 07:47:40.418797 7443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:40.435411 7443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:40.437498 7443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:40.442233 7443 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.252:36421
I20260621 07:47:40.442260 7698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.252:36421 every 8 connection(s)
I20260621 07:47:40.442813 7443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/master-2/data/info.pb
I20260621 07:47:40.444283 7699 sys_catalog.cc:263] Verifying existing consensus state
I20260621 07:47:40.445178 7699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d: Bootstrap starting.
I20260621 07:47:40.452760 7699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 07:47:40.453004 7699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d: Bootstrap complete.
I20260621 07:47:40.453428 7699 raft_consensus.cc:348] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: false } } peers { permanent_uuid: "301f9ea85622455aaa68534f71280c4d" member_type: NON_VOTER last_known_addr { host: "127.0.192.252" port: 36421 } attrs { promote: true } }
I20260621 07:47:40.453534 7699 raft_consensus.cc:729] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 301f9ea85622455aaa68534f71280c4d, State: Initialized, Role: LEARNER
I20260621 07:47:40.453632 7699 consensus_queue.cc:260] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: false } } peers { permanent_uuid: "301f9ea85622455aaa68534f71280c4d" member_type: NON_VOTER last_known_addr { host: "127.0.192.252" port: 36421 } attrs { promote: true } }
I20260621 07:47:40.453918 7705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: false } } peers { permanent_uuid: "301f9ea85622455aaa68534f71280c4d" member_type: NON_VOTER last_known_addr { host: "127.0.192.252" port: 36421 } attrs { promote: true } } }
I20260621 07:47:40.454073 7705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [sys.catalog]: This master's current role is: LEARNER
I20260621 07:47:40.454077 7699 sys_catalog.cc:564] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:47:40.457391 7716 catalog_manager.cc:1259] Loaded cluster ID: 2feba99244ef42a5b71acfd2c5120240
I20260621 07:47:40.457489 7716 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d: loading cluster ID for follower catalog manager: success
W20260621 07:47:40.457762 7616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:40.457989 7616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:40.458060 7616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260621 07:47:40.459444 7716 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d: acquiring CA information for follower catalog manager: success
I20260621 07:47:40.460716 7716 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260621 07:47:40.461946 7616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:40.462013 7616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:40.462092 7616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.193
I20260621 07:47:40.466452 7616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:39257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/oryx_127.0.192.193.keytab
--principal=oryx/127.0.192.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.193:45293
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.192.193
--webserver_port=39811
--tserver_master_addrs=127.0.192.254:40701,127.0.192.253:43105,127.0.192.252:36421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.193
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:40.467298 7616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:40.468184 7616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:40.475880 7721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:40.476055 7616 server_base.cc:1034] running on GCE node
W20260621 07:47:40.475944 7718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:40.475915 7719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:40.476485 7616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:40.476991 7616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:40.478193 7616 hybrid_clock.cc:648] HybridClock initialized: now 1782028060478163 us; error 36 us; skew 500 ppm
Jun 21 07:47:40 dist-test-slave-lzx3 krb5kdc[6826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028060, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.192.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:40.480700 7616 init.cc:377] Logged in from keytab as oryx/127.0.192.193@KRBTEST.COM (short username oryx)
I20260621 07:47:40.481371 7616 webserver.cc:460] Webserver started at http://127.0.192.193:39811/ using document root <none> and password file <none>
I20260621 07:47:40.481999 7616 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:40.482095 7616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:40.486444 7616 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:47:40.488173 7728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:40.489272 7616 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:40.489384 7616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-0/wal
uuid: "efd8a37d6ac44bb4b4ef2f0e61cdb11c"
format_stamp: "Formatted at 2026-06-21 07:47:39 on dist-test-slave-lzx3"
server_key: "02e7d2242d7c16d9478f24900dda7d17"
server_key_iv: "9694162935e13f6ddc90bb5cb9c71609"
server_key_version: "encryptionkey@0"
I20260621 07:47:40.489801 7616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:40.516314 7616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:40.519101 7616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:40.519277 7616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:40.519781 7616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:47:40.520486 7616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:47:40.520547 7616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:40.520620 7616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:47:40.520658 7616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:40.529192 7616 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.193:45293
I20260621 07:47:40.529254 7841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.193:45293 every 8 connection(s)
I20260621 07:47:40.530026 7616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
I20260621 07:47:40.534202 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 7616
I20260621 07:47:40.534402 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 7067
Jun 21 07:47:40 dist-test-slave-lzx3 krb5kdc[6826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028060, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.192.193@KRBTEST.COM for oryx/127.0.192.253@KRBTEST.COM
Jun 21 07:47:40 dist-test-slave-lzx3 krb5kdc[6826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028060, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.192.193@KRBTEST.COM for oryx/127.0.192.252@KRBTEST.COM
I20260621 07:47:40.541419 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.194:40815
--local_ip_for_outbound_sockets=127.0.192.194
--tserver_master_addrs=127.0.192.254:40701
--webserver_port=43591
--webserver_interface=127.0.192.194
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/oryx_127.0.192.194.keytab
--principal=oryx/127.0.192.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:39257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.192.254:40701,127.0.192.253:43105
--tserver_master_addrs=127.0.192.254:40701,127.0.192.253:43105,127.0.192.252:36421 with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 07:47:40 dist-test-slave-lzx3 krb5kdc[6826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028060, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.192.193@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM
I20260621 07:47:40.545802 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.531585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:38833 (local address 127.0.192.252:36421)
0621 07:47:40.531789 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:40.531793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:40.532390 (+ 597us) server_negotiation.cc:407] Connection header received
0621 07:47:40.533351 (+ 961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:40.533353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:40.533404 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:40.533493 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:40.536024 (+ 2531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.537021 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.538105 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.538493 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.538579 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.541630 (+ 3051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:40.541634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:40.541637 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:40.541663 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:40.543406 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.544035 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.544040 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.544044 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.544124 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.544546 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.544550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.544552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.544788 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:40.544862 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:40.545499 (+ 637us) server_negotiation.cc:299] Negotiation successful
0621 07:47:40.545619 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":80,"threads_started":1}
I20260621 07:47:40.546576 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.531514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:60427 (local address 127.0.192.254:40701)
0621 07:47:40.531639 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:40.531643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:40.532392 (+ 749us) server_negotiation.cc:407] Connection header received
0621 07:47:40.533410 (+ 1018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:40.533419 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:40.533488 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:40.533554 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:40.537266 (+ 3712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.538185 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.539397 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.539800 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.539907 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.542256 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:40.542262 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:40.542268 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:40.542302 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:40.544069 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.544546 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.544552 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.544554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.544658 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.544986 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.544992 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.544996 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.545216 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:40.545310 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:40.546261 (+ 951us) server_negotiation.cc:299] Negotiation successful
0621 07:47:40.546382 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260621 07:47:40.546597 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.532875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:55427 (local address 127.0.192.253:43105)
0621 07:47:40.532968 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:40.532971 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:40.533126 (+ 155us) server_negotiation.cc:407] Connection header received
0621 07:47:40.533327 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:40.533330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:40.533379 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:40.533441 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:40.534995 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.535676 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.537209 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.537571 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.537648 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.543101 (+ 5453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:40.543105 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:40.543110 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:40.543139 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:40.544925 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.545359 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.545362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.545364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.545442 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.545723 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.545726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.545727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.545909 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:40.545977 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:40.546365 (+ 388us) server_negotiation.cc:299] Negotiation successful
0621 07:47:40.546462 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260621 07:47:40.546838 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.531878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:36421 (local address 127.0.192.193:38833)
0621 07:47:40.532268 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:40.532311 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:40.533156 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:40.533686 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:40.533692 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:40.534154 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:40.535912 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.535923 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.537138 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.537144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.537964 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.537972 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.538734 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.538740 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.538835 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.539229 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:40.539242 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:40.541433 (+ 2191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:40.543557 (+ 2124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:40.543565 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:40.543581 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:40.543911 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:40.544237 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:40.544242 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:40.544245 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:40.544431 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:40.544950 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:40.544957 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:40.545248 (+ 291us) client_negotiation.cc:769] Sending connection context
0621 07:47:40.545924 (+ 676us) client_negotiation.cc:240] Negotiation successful
0621 07:47:40.546070 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":85,"spinlock_wait_cycles":7040,"thread_start_us":108,"threads_started":1}
I20260621 07:47:40.547026 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.532753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.253:43105 (local address 127.0.192.193:55427)
0621 07:47:40.533051 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:40.533060 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:40.533155 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:40.533684 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:40.533690 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:40.534120 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:40.534828 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.534848 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.535817 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.535819 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.537085 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.537096 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.537818 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.537821 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.538013 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.538614 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:40.538642 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:40.542955 (+ 4313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:40.545044 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:40.545046 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:40.545048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:40.545253 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:40.545556 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:40.545558 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:40.545560 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:40.545632 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:40.546100 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:40.546103 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:40.546188 (+ 85us) client_negotiation.cc:769] Sending connection context
0621 07:47:40.546747 (+ 559us) client_negotiation.cc:240] Negotiation successful
0621 07:47:40.546866 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":25,"spinlock_wait_cycles":3456,"thread_start_us":79,"threads_started":1}
I20260621 07:47:40.547977 7842 heartbeater.cc:347] Connected to a master server at 127.0.192.252:36421
I20260621 07:47:40.548204 7842 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:40.548763 7842 heartbeater.cc:511] Master 127.0.192.252:36421 requested a full tablet report, sending...
I20260621 07:47:40.549264 7844 heartbeater.cc:347] Connected to a master server at 127.0.192.253:43105
I20260621 07:47:40.549340 7844 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:40.549508 7844 heartbeater.cc:511] Master 127.0.192.253:43105 requested a full tablet report, sending...
I20260621 07:47:40.550081 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.531878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:40701 (local address 127.0.192.193:60427)
0621 07:47:40.532269 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:40.532311 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:40.533157 (+ 846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:40.533686 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:40.533692 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:40.534162 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:40.537131 (+ 2969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.537144 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.538318 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.538324 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.539273 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.539285 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.539939 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.539942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.540034 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.540431 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:40.540448 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:40.541825 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:40.544177 (+ 2352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:40.544182 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:40.544187 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:40.544439 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:40.544754 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:40.544759 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:40.544764 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:40.544849 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:40.545353 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:40.545358 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:40.545455 (+ 97us) client_negotiation.cc:769] Sending connection context
0621 07:47:40.545569 (+ 114us) client_negotiation.cc:240] Negotiation successful
0621 07:47:40.545879 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":2588,"spinlock_wait_cycles":31488,"thread_start_us":88,"threads_started":1}
I20260621 07:47:40.550187 7663 ts_manager.cc:194] Registered new tserver with Master: efd8a37d6ac44bb4b4ef2f0e61cdb11c (127.0.192.193:45293)
I20260621 07:47:40.551100 7471 ts_manager.cc:194] Registered new tserver with Master: efd8a37d6ac44bb4b4ef2f0e61cdb11c (127.0.192.193:45293)
I20260621 07:47:40.551419 7843 heartbeater.cc:347] Connected to a master server at 127.0.192.254:40701
I20260621 07:47:40.551498 7843 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:40.551687 7843 heartbeater.cc:511] Master 127.0.192.254:40701 requested a full tablet report, sending...
I20260621 07:47:40.552445 6871 ts_manager.cc:194] Re-registered known tserver with Master: efd8a37d6ac44bb4b4ef2f0e61cdb11c (127.0.192.193:45293)
I20260621 07:47:40.553226 6871 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.192.193@KRBTEST.COM'} at 127.0.192.193:60427
W20260621 07:47:40.646394 7850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:40.646665 7850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:40.646706 7850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:40.650578 7850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:40.650621 7850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:40.650669 7850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.194
I20260621 07:47:40.655545 7850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:39257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/oryx_127.0.192.194.keytab
--principal=oryx/127.0.192.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.194:40815
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.192.194
--webserver_port=43591
--tserver_master_addrs=127.0.192.254:40701,127.0.192.253:43105,127.0.192.252:36421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.194
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:40.656387 7850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:40.657214 7850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:40.665630 7856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:40.665670 7858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:40.665784 7850 server_base.cc:1034] running on GCE node
W20260621 07:47:40.665632 7855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:40.666245 7850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:40.666848 7850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:40.668016 7850 hybrid_clock.cc:648] HybridClock initialized: now 1782028060668017 us; error 34 us; skew 500 ppm
Jun 21 07:47:40 dist-test-slave-lzx3 krb5kdc[6826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028060, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.192.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:40.670361 7850 init.cc:377] Logged in from keytab as oryx/127.0.192.194@KRBTEST.COM (short username oryx)
I20260621 07:47:40.671094 7850 webserver.cc:460] Webserver started at http://127.0.192.194:43591/ using document root <none> and password file <none>
I20260621 07:47:40.671571 7850 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:40.671657 7850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:40.674825 7850 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260621 07:47:40.676744 7865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:40.677824 7850 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:40.677950 7850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-1/wal
uuid: "65c308b30fa54aaba7bd5203d9b258a1"
format_stamp: "Formatted at 2026-06-21 07:47:39 on dist-test-slave-lzx3"
server_key: "1b47ac7a137f994bcfcff826d628b6fb"
server_key_iv: "35de321f802bd747ef0abb44f73025af"
server_key_version: "encryptionkey@0"
I20260621 07:47:40.678426 7850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:40.694571 7850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:40.696975 7850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:40.697180 7850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:40.697777 7850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:47:40.698557 7850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:47:40.698640 7850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:40.698702 7850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:47:40.698730 7850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:40.706247 7850 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.194:40815
I20260621 07:47:40.706280 7978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.194:40815 every 8 connection(s)
I20260621 07:47:40.707021 7850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
I20260621 07:47:40.707491 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 7850
I20260621 07:47:40.707679 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 7201
I20260621 07:47:40.715530 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.195:43223
--local_ip_for_outbound_sockets=127.0.192.195
--tserver_master_addrs=127.0.192.254:40701
--webserver_port=44405
--webserver_interface=127.0.192.195
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/oryx_127.0.192.195.keytab
--principal=oryx/127.0.192.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:39257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.192.254:40701,127.0.192.253:43105
--tserver_master_addrs=127.0.192.254:40701,127.0.192.253:43105,127.0.192.252:36421 with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 07:47:40 dist-test-slave-lzx3 krb5kdc[6826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028060, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.192.194@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM
Jun 21 07:47:40 dist-test-slave-lzx3 krb5kdc[6826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028060, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.192.194@KRBTEST.COM for oryx/127.0.192.253@KRBTEST.COM
Jun 21 07:47:40 dist-test-slave-lzx3 krb5kdc[6826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028060, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.192.194@KRBTEST.COM for oryx/127.0.192.252@KRBTEST.COM
I20260621 07:47:40.723671 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.710071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:35975 (local address 127.0.192.254:40701)
0621 07:47:40.710710 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:40.710715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:40.711100 (+ 385us) server_negotiation.cc:407] Connection header received
0621 07:47:40.711618 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:40.711624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:40.711700 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:40.711781 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:40.713159 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.714024 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.715860 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.716249 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.716323 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.719167 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:40.719171 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:40.719174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:40.719200 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:40.720872 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.721474 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.721479 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.721484 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.721568 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.722131 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.722137 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.722145 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.722434 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:40.722548 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:40.723304 (+ 756us) server_negotiation.cc:299] Negotiation successful
0621 07:47:40.723442 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558}
I20260621 07:47:40.723737 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.708882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:56421 (local address 127.0.192.252:36421)
0621 07:47:40.709112 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:40.709116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:40.710567 (+ 1451us) server_negotiation.cc:407] Connection header received
0621 07:47:40.711315 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:40.711321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:40.711381 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:40.711480 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:40.713189 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.713961 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.715859 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.716197 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.716287 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.720292 (+ 4005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:40.720295 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:40.720298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:40.720328 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:40.721812 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.722333 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.722336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.722338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.722408 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.722922 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.722925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.722927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.723131 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:40.723217 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:40.723508 (+ 291us) server_negotiation.cc:299] Negotiation successful
0621 07:47:40.723614 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165}
I20260621 07:47:40.724381 7985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.709090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.253:43105 (local address 127.0.192.194:42367)
0621 07:47:40.710107 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:40.710155 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:40.711364 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:40.711692 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:40.711695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:40.712233 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:40.712983 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.713004 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.714127 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.714130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.715847 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.715854 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.716458 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.716461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.716739 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.717472 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:40.717491 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:40.719143 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:40.721830 (+ 2687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:40.721835 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:40.721840 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:40.722181 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:40.722503 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:40.722506 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:40.722507 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:40.722604 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:40.723077 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:40.723080 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:40.723162 (+ 82us) client_negotiation.cc:769] Sending connection context
0621 07:47:40.723268 (+ 106us) client_negotiation.cc:240] Negotiation successful
0621 07:47:40.723561 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":827,"mutex_wait_us":466,"thread_start_us":617,"threads_started":1}
I20260621 07:47:40.724869 7984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.709091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:36421 (local address 127.0.192.194:56421)
0621 07:47:40.710472 (+ 1381us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:40.710484 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:40.711119 (+ 635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:40.711632 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:40.711659 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:40.712233 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:40.712984 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.712999 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.714127 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.714130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.715730 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.715740 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.716428 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.716432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.716713 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.718514 (+ 1801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:40.718529 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:40.720140 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:40.721912 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:40.721917 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:40.721921 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:40.722233 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:40.722511 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:40.722514 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:40.722515 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:40.722819 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:40.723259 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:40.723262 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:40.723343 (+ 81us) client_negotiation.cc:769] Sending connection context
0621 07:47:40.723409 (+ 66us) client_negotiation.cc:240] Negotiation successful
0621 07:47:40.723976 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1307,"mutex_wait_us":959,"spinlock_wait_cycles":23552,"thread_start_us":138,"threads_started":1}
I20260621 07:47:40.725015 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.708766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:42367 (local address 127.0.192.253:43105)
0621 07:47:40.709024 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:40.709029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:40.710251 (+ 1222us) server_negotiation.cc:407] Connection header received
0621 07:47:40.711446 (+ 1195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:40.711449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:40.711499 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:40.711566 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:40.713161 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.713961 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.715970 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.716286 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.716341 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.719273 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:40.719276 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:40.719278 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:40.719301 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:40.721713 (+ 2412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.722305 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.722310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.722315 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.722398 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.722712 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.722715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.722717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.722939 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:40.723035 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:40.724739 (+ 1704us) server_negotiation.cc:299] Negotiation successful
0621 07:47:40.724848 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260621 07:47:40.725246 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.709751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:40701 (local address 127.0.192.194:35975)
0621 07:47:40.710992 (+ 1241us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:40.711017 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:40.711119 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:40.711917 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:40.711923 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:40.712245 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:40.712979 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.713000 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.714163 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.714169 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.715730 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.715743 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.716519 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.716524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.716752 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.717259 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:40.717272 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:40.718905 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:40.721001 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:40.721009 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:40.721034 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:40.721346 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:40.721702 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:40.721707 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:40.721710 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:40.721881 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:40.722656 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:40.722665 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:40.722977 (+ 312us) client_negotiation.cc:769] Sending connection context
0621 07:47:40.724261 (+ 1284us) client_negotiation.cc:240] Negotiation successful
0621 07:47:40.724413 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1181,"mutex_wait_us":276,"spinlock_wait_cycles":12544,"thread_start_us":79,"threads_started":1}
I20260621 07:47:40.725667 7979 heartbeater.cc:347] Connected to a master server at 127.0.192.252:36421
I20260621 07:47:40.725764 7981 heartbeater.cc:347] Connected to a master server at 127.0.192.253:43105
I20260621 07:47:40.725878 7981 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:40.726068 7980 heartbeater.cc:347] Connected to a master server at 127.0.192.254:40701
I20260621 07:47:40.726132 7980 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:40.725879 7979 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:40.726393 7979 heartbeater.cc:511] Master 127.0.192.252:36421 requested a full tablet report, sending...
I20260621 07:47:40.726395 7981 heartbeater.cc:511] Master 127.0.192.253:43105 requested a full tablet report, sending...
I20260621 07:47:40.726397 7980 heartbeater.cc:511] Master 127.0.192.254:40701 requested a full tablet report, sending...
I20260621 07:47:40.727392 7471 ts_manager.cc:194] Registered new tserver with Master: 65c308b30fa54aaba7bd5203d9b258a1 (127.0.192.194:40815)
I20260621 07:47:40.727483 6871 ts_manager.cc:194] Re-registered known tserver with Master: 65c308b30fa54aaba7bd5203d9b258a1 (127.0.192.194:40815)
I20260621 07:47:40.727941 7663 ts_manager.cc:194] Registered new tserver with Master: 65c308b30fa54aaba7bd5203d9b258a1 (127.0.192.194:40815)
I20260621 07:47:40.728359 6871 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.192.194@KRBTEST.COM'} at 127.0.192.194:35975
I20260621 07:47:40.795825 7428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.791878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:36421 (local address 127.0.0.1:53852)
0621 07:47:40.792093 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:40.792139 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:40.792266 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:40.792651 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:40.792653 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:40.792669 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:40.792746 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.792750 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.793877 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.793879 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.794599 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.794607 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.795534 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.795537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.795603 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.795621 (+ 18us) client_negotiation.cc:769] Sending connection context
0621 07:47:40.795669 (+ 48us) client_negotiation.cc:240] Negotiation successful
0621 07:47:40.795724 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127}
I20260621 07:47:40.795925 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.791994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53852 (local address 127.0.192.252:36421)
0621 07:47:40.792159 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:40.792164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:40.792188 (+ 24us) server_negotiation.cc:407] Connection header received
0621 07:47:40.792388 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:40.792393 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:40.792481 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:40.792639 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:40.792851 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.793761 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.794732 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.795386 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.795526 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.795677 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:40.795721 (+ 44us) server_negotiation.cc:299] Negotiation successful
0621 07:47:40.795799 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 07:47:40.796396 7673 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260621 07:47:40.796684 7705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8072ff3594143ca8e36eb920eae693c. Latest consensus state: current_term: 1 leader_uuid: "c8072ff3594143ca8e36eb920eae693c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: false } } peers { permanent_uuid: "301f9ea85622455aaa68534f71280c4d" member_type: NON_VOTER last_known_addr { host: "127.0.192.252" port: 36421 } attrs { promote: true } } }
I20260621 07:47:40.796818 7705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [sys.catalog]: This master's current role is: LEARNER
W20260621 07:47:40.819595 7986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:40.819795 7986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:40.819857 7986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:40.823163 7986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:40.823230 7986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:40.823309 7986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.195
I20260621 07:47:40.827600 7986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:39257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/krb5kdc/oryx_127.0.192.195.keytab
--principal=oryx/127.0.192.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.195:43223
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.192.195
--webserver_port=44405
--tserver_master_addrs=127.0.192.254:40701,127.0.192.253:43105,127.0.192.252:36421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.195
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:40.828452 7986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:40.829313 7986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 07:47:40.837055 7986 server_base.cc:1034] running on GCE node
W20260621 07:47:40.837028 7991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:40.837042 7992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:40.837023 7994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:40.837522 7986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:40.838097 7986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:40.839275 7986 hybrid_clock.cc:648] HybridClock initialized: now 1782028060839256 us; error 29 us; skew 500 ppm
Jun 21 07:47:40 dist-test-slave-lzx3 krb5kdc[6826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028060, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.192.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:40.841737 7986 init.cc:377] Logged in from keytab as oryx/127.0.192.195@KRBTEST.COM (short username oryx)
I20260621 07:47:40.842430 7986 webserver.cc:460] Webserver started at http://127.0.192.195:44405/ using document root <none> and password file <none>
I20260621 07:47:40.842936 7986 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:40.843034 7986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:40.846041 7986 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260621 07:47:40.847644 8001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:40.848776 7986 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:40.848881 7986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-2/wal
uuid: "eecd298bb09548e0b4f4edffda7cbe91"
format_stamp: "Formatted at 2026-06-21 07:47:39 on dist-test-slave-lzx3"
server_key: "6f0c5d196a0a1d48dbab76dd241e8658"
server_key_iv: "e9ed6255ccf78561234211f9556b0205"
server_key_version: "encryptionkey@0"
I20260621 07:47:40.849285 7986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:40.862002 7986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:40.864709 7986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:40.864871 7986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:40.865324 7986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:47:40.866014 7986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:47:40.866076 7986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:40.866133 7986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:47:40.866165 7986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:40.873953 7986 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.195:43223
I20260621 07:47:40.873973 8114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.195:43223 every 8 connection(s)
I20260621 07:47:40.874694 7986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
I20260621 07:47:40.882300 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 7986
Jun 21 07:47:40 dist-test-slave-lzx3 krb5kdc[6826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028060, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.192.195@KRBTEST.COM for oryx/127.0.192.252@KRBTEST.COM
Jun 21 07:47:40 dist-test-slave-lzx3 krb5kdc[6826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028060, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.192.195@KRBTEST.COM for oryx/127.0.192.253@KRBTEST.COM
Jun 21 07:47:40 dist-test-slave-lzx3 krb5kdc[6826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028060, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.192.195@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM
I20260621 07:47:40.889248 8121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.877302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:36421 (local address 127.0.192.195:49583)
0621 07:47:40.877573 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:40.877586 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:40.878093 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:40.878603 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:40.878609 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:40.878990 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:40.879408 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.879419 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.880091 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.880093 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.880888 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.880895 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.881563 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.881566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.881796 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.882351 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:40.882372 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:40.883955 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:40.885837 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:40.885842 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:40.885854 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:40.886811 (+ 957us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:40.887108 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:40.887111 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:40.887112 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:40.887245 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:40.887677 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:40.887683 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:40.887930 (+ 247us) client_negotiation.cc:769] Sending connection context
0621 07:47:40.888120 (+ 190us) client_negotiation.cc:240] Negotiation successful
0621 07:47:40.888380 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":36,"spinlock_wait_cycles":23296,"thread_start_us":105,"threads_started":1}
I20260621 07:47:40.889945 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.876355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:49583 (local address 127.0.192.252:36421)
0621 07:47:40.876478 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:40.876482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:40.877682 (+ 1200us) server_negotiation.cc:407] Connection header received
0621 07:47:40.878282 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:40.878285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:40.878343 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:40.878434 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:40.879529 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.879966 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.880990 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.881278 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.881379 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.884119 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:40.884124 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:40.884128 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:40.884154 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:40.885724 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.886908 (+ 1184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.886911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.886913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.887019 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.887332 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.887334 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.887336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.887545 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:40.887629 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:40.889681 (+ 2052us) server_negotiation.cc:299] Negotiation successful
0621 07:47:40.889807 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 07:47:40.890724 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.876355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:52269 (local address 127.0.192.254:40701)
0621 07:47:40.876464 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:40.876469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:40.877664 (+ 1195us) server_negotiation.cc:407] Connection header received
0621 07:47:40.878297 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:40.878300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:40.878350 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:40.878401 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:40.879546 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.880218 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.881372 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.881673 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.881768 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.886761 (+ 4993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:40.886763 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:40.886766 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:40.886783 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:40.888845 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.889277 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.889280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.889282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.889358 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.889872 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.889875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.889877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.890111 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:40.890189 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:40.890495 (+ 306us) server_negotiation.cc:299] Negotiation successful
0621 07:47:40.890591 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260621 07:47:40.890952 8115 heartbeater.cc:347] Connected to a master server at 127.0.192.252:36421
I20260621 07:47:40.891155 8115 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:40.891212 8120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.876682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:40701 (local address 127.0.192.195:52269)
0621 07:47:40.877567 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:40.877580 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:40.878083 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:40.878602 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:40.878610 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:40.878987 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:40.879421 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.879428 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.880340 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.880343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.881264 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.881273 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.881769 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.881775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.881890 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.883214 (+ 1324us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:40.883232 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:40.886639 (+ 3407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:40.888954 (+ 2315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:40.888956 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:40.888958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:40.889179 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:40.889454 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:40.889457 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:40.889458 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:40.889545 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:40.890246 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:40.890249 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:40.890331 (+ 82us) client_negotiation.cc:769] Sending connection context
0621 07:47:40.890908 (+ 577us) client_negotiation.cc:240] Negotiation successful
0621 07:47:40.891055 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"mutex_wait_us":190,"spinlock_wait_cycles":49408,"thread_start_us":126,"threads_started":1}
I20260621 07:47:40.891587 8115 heartbeater.cc:511] Master 127.0.192.252:36421 requested a full tablet report, sending...
I20260621 07:47:40.891686 8116 heartbeater.cc:347] Connected to a master server at 127.0.192.254:40701
I20260621 07:47:40.891752 8116 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:40.892148 8116 heartbeater.cc:511] Master 127.0.192.254:40701 requested a full tablet report, sending...
I20260621 07:47:40.892364 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.876203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:47287 (local address 127.0.192.253:43105)
0621 07:47:40.876301 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:40.876305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:40.877348 (+ 1043us) server_negotiation.cc:407] Connection header received
0621 07:47:40.878297 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:40.878300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:40.878350 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:40.878402 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:40.879547 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.880218 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.881387 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.881715 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.881800 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.884765 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:40.884769 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:40.884772 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:40.884795 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:40.886405 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.891225 (+ 4820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.891229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.891230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.891312 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.891565 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.891567 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.891570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.891745 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:40.891841 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:40.892154 (+ 313us) server_negotiation.cc:299] Negotiation successful
0621 07:47:40.892244 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260621 07:47:40.892371 8119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.876686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.253:43105 (local address 127.0.192.195:47287)
0621 07:47:40.877181 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:40.877237 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:40.878083 (+ 846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:40.878602 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:40.878610 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:40.878980 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:40.879422 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.879428 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.880340 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.880343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.881266 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.881273 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.881913 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.881915 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.882004 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.882685 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:40.882698 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:40.884345 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:40.890883 (+ 6538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:40.890887 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:40.890888 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:40.891119 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:40.891402 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:40.891405 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:40.891407 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:40.891484 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:40.891904 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:40.891907 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:40.891991 (+ 84us) client_negotiation.cc:769] Sending connection context
0621 07:47:40.892097 (+ 106us) client_negotiation.cc:240] Negotiation successful
0621 07:47:40.892207 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":191,"spinlock_wait_cycles":3328,"thread_start_us":569,"threads_started":1}
I20260621 07:47:40.892618 7663 ts_manager.cc:194] Registered new tserver with Master: eecd298bb09548e0b4f4edffda7cbe91 (127.0.192.195:43223)
I20260621 07:47:40.892803 6871 ts_manager.cc:194] Re-registered known tserver with Master: eecd298bb09548e0b4f4edffda7cbe91 (127.0.192.195:43223)
I20260621 07:47:40.893484 8117 heartbeater.cc:347] Connected to a master server at 127.0.192.253:43105
I20260621 07:47:40.893566 8117 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:40.893698 8117 heartbeater.cc:511] Master 127.0.192.253:43105 requested a full tablet report, sending...
I20260621 07:47:40.894516 7471 ts_manager.cc:194] Registered new tserver with Master: eecd298bb09548e0b4f4edffda7cbe91 (127.0.192.195:43223)
I20260621 07:47:40.895201 6871 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.192.195@KRBTEST.COM'} at 127.0.192.195:52269
I20260621 07:47:40.896754 8127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.884175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40244 (local address 127.0.192.253:43105)
0621 07:47:40.885836 (+ 1661us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:40.885842 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:40.885856 (+ 14us) server_negotiation.cc:407] Connection header received
0621 07:47:40.885891 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:40.885893 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:40.885935 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:40.886045 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:40.886808 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.887444 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.888620 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.888926 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.889008 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.891513 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:40.891519 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:40.891521 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:40.891545 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:40.894085 (+ 2540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.894478 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.894482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.894483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.894564 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.895787 (+ 1223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.895790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.895792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.896003 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:40.896122 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:40.896516 (+ 394us) server_negotiation.cc:299] Negotiation successful
0621 07:47:40.896631 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1590,"thread_start_us":65,"threads_started":1}
I20260621 07:47:40.900355 8130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.884788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53856 (local address 127.0.192.252:36421)
0621 07:47:40.885327 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:40.885332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:40.890241 (+ 4909us) server_negotiation.cc:407] Connection header received
0621 07:47:40.893064 (+ 2823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:40.893067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:40.893105 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:40.893154 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:40.893842 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.894488 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.895260 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.895610 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.895704 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.897211 (+ 1507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:40.897214 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:40.897216 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:40.897237 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:40.898865 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.899331 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.899335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.899336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.899411 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.899652 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.899655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.899656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.899796 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:40.899909 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:40.900111 (+ 202us) server_negotiation.cc:299] Negotiation successful
0621 07:47:40.900209 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":68,"threads_started":1}
I20260621 07:47:40.900524 8126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.884017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34206 (local address 127.0.192.254:40701)
0621 07:47:40.886576 (+ 2559us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:40.886581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:40.886594 (+ 13us) server_negotiation.cc:407] Connection header received
0621 07:47:40.886625 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:40.886627 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:40.886666 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:40.886739 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:40.887688 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.888953 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.890374 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.890684 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.893915 (+ 3231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.895588 (+ 1673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:40.895592 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:40.895594 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:40.895616 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:40.898832 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.899367 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.899371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.899373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.899474 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:40.899743 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:40.899746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:40.899748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:40.899902 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:40.900019 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:40.900261 (+ 242us) server_negotiation.cc:299] Negotiation successful
0621 07:47:40.900365 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2471,"thread_start_us":51,"threads_started":1}
I20260621 07:47:40.902735 6871 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34206:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:47:40.905223 6871 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:47:40.907352 8133 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8072ff3594143ca8e36eb920eae693c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: false } } peers { permanent_uuid: "301f9ea85622455aaa68534f71280c4d" member_type: NON_VOTER last_known_addr { host: "127.0.192.252" port: 36421 } attrs { promote: true } } }
I20260621 07:47:40.907500 8133 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:40.908928 8133 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c: attempting to promote NON_VOTER 301f9ea85622455aaa68534f71280c4d to VOTER
I20260621 07:47:40.909252 8133 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: false } } peers { permanent_uuid: "301f9ea85622455aaa68534f71280c4d" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 36421 } attrs { promote: false } }
I20260621 07:47:40.910190 7673 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [term 1 LEARNER]: Refusing update from remote peer c8072ff3594143ca8e36eb920eae693c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260621 07:47:40.910205 7481 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [term 1 FOLLOWER]: Refusing update from remote peer c8072ff3594143ca8e36eb920eae693c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260621 07:47:40.910539 7617 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [LEADER]: Connected to new peer: Peer: permanent_uuid: "301f9ea85622455aaa68534f71280c4d" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 36421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260621 07:47:40.910694 8133 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260621 07:47:40.912277 8134 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 301f9ea85622455aaa68534f71280c4d (127.0.192.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: false } } peers { permanent_uuid: "301f9ea85622455aaa68534f71280c4d" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 36421 } attrs { promote: false } } }
I20260621 07:47:40.912519 7481 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 301f9ea85622455aaa68534f71280c4d (127.0.192.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: false } } peers { permanent_uuid: "301f9ea85622455aaa68534f71280c4d" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 36421 } attrs { promote: false } } }
I20260621 07:47:40.912612 7673 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 301f9ea85622455aaa68534f71280c4d (127.0.192.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: false } } peers { permanent_uuid: "301f9ea85622455aaa68534f71280c4d" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 36421 } attrs { promote: false } } }
I20260621 07:47:40.914070 7705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8072ff3594143ca8e36eb920eae693c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: false } } peers { permanent_uuid: "301f9ea85622455aaa68534f71280c4d" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 36421 } attrs { promote: false } } }
I20260621 07:47:40.914175 7705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:47:40.914377 8133 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8072ff3594143ca8e36eb920eae693c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: false } } peers { permanent_uuid: "301f9ea85622455aaa68534f71280c4d" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 36421 } attrs { promote: false } } }
I20260621 07:47:40.914486 8133 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:40.914563 8134 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8072ff3594143ca8e36eb920eae693c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: false } } peers { permanent_uuid: "301f9ea85622455aaa68534f71280c4d" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 36421 } attrs { promote: false } } }
I20260621 07:47:40.914678 8134 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:40.915359 8142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8072ff3594143ca8e36eb920eae693c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: false } } peers { permanent_uuid: "301f9ea85622455aaa68534f71280c4d" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 36421 } attrs { promote: false } } }
I20260621 07:47:40.915495 8142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:47:40.916031 7617 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c8072ff3594143ca8e36eb920eae693c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: false } } peers { permanent_uuid: "301f9ea85622455aaa68534f71280c4d" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 36421 } attrs { promote: false } } }
I20260621 07:47:40.916142 7617 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:40.925359 7428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.920238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:40815 (local address 127.0.0.1:41024)
0621 07:47:40.920382 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:40.920396 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:40.920485 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:40.921548 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:40.921551 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:40.921572 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:40.921663 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.921672 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.922978 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.922981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.923957 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.923980 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.924950 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.924954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.925078 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.925099 (+ 21us) client_negotiation.cc:769] Sending connection context
0621 07:47:40.925181 (+ 82us) client_negotiation.cc:240] Negotiation successful
0621 07:47:40.925247 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":22}
I20260621 07:47:40.925508 8144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.920363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41024 (local address 127.0.192.194:40815)
0621 07:47:40.921037 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:40.921051 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:40.921082 (+ 31us) server_negotiation.cc:407] Connection header received
0621 07:47:40.921149 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:40.921153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:40.921296 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:40.921841 (+ 545us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:40.922013 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.922855 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.924131 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.924818 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.924952 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.925097 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:40.925211 (+ 114us) server_negotiation.cc:299] Negotiation successful
0621 07:47:40.925290 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":92,"threads_started":1}
I20260621 07:47:40.926548 8147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.920439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.195:43223 (local address 127.0.0.1:43310)
0621 07:47:40.921978 (+ 1539us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:40.921992 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:40.922083 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:40.922536 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:40.922538 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:40.922553 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:40.922624 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.922629 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.923995 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.923999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.925137 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.925147 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.926211 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.926215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.926304 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.926320 (+ 16us) client_negotiation.cc:769] Sending connection context
0621 07:47:40.926372 (+ 52us) client_negotiation.cc:240] Negotiation successful
0621 07:47:40.926417 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1481,"mutex_wait_us":59,"thread_start_us":210,"threads_started":1}
I20260621 07:47:40.926662 8145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.920971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:45293 (local address 127.0.0.1:51748)
0621 07:47:40.922265 (+ 1294us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:40.922277 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:40.922356 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:40.922725 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:40.922728 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:40.922744 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:40.922822 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.922827 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.923657 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.923661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.925137 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:40.925147 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.926319 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.926322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:40.926397 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.926417 (+ 20us) client_negotiation.cc:769] Sending connection context
0621 07:47:40.926486 (+ 69us) client_negotiation.cc:240] Negotiation successful
0621 07:47:40.926555 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1240,"mutex_wait_us":38,"thread_start_us":498,"threads_started":1}
I20260621 07:47:40.926776 8148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.921522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51748 (local address 127.0.192.193:45293)
0621 07:47:40.921900 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:40.921912 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:40.922364 (+ 452us) server_negotiation.cc:407] Connection header received
0621 07:47:40.922439 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:40.922443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:40.922598 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:40.922728 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:40.922931 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.923555 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.925267 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.926001 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.926118 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.926261 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:40.926541 (+ 280us) server_negotiation.cc:299] Negotiation successful
0621 07:47:40.926642 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":87,"threads_started":1}
I20260621 07:47:40.928097 7913 tablet_service.cc:1467] Processing CreateTablet for tablet fbe595e147b4459ab774f1129b6edc46 (DEFAULT_TABLE table=test-table [id=4229b48b6257473ca7743a7244aa4793]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:47:40.929117 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:40.920882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43310 (local address 127.0.192.195:43223)
0621 07:47:40.921903 (+ 1021us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:40.921912 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:40.922070 (+ 158us) server_negotiation.cc:407] Connection header received
0621 07:47:40.922174 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:40.922177 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:40.922292 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:40.922424 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:40.922956 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.923881 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.925267 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:40.926076 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:40.928811 (+ 2735us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:40.928935 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:40.928977 (+ 42us) server_negotiation.cc:299] Negotiation successful
0621 07:47:40.929031 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":960,"thread_start_us":517,"threads_started":1}
I20260621 07:47:40.929157 7913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbe595e147b4459ab774f1129b6edc46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:40.929277 7776 tablet_service.cc:1467] Processing CreateTablet for tablet fbe595e147b4459ab774f1129b6edc46 (DEFAULT_TABLE table=test-table [id=4229b48b6257473ca7743a7244aa4793]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:47:40.930105 7776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbe595e147b4459ab774f1129b6edc46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:40.930879 8049 tablet_service.cc:1467] Processing CreateTablet for tablet fbe595e147b4459ab774f1129b6edc46 (DEFAULT_TABLE table=test-table [id=4229b48b6257473ca7743a7244aa4793]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:47:40.931537 8049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbe595e147b4459ab774f1129b6edc46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:40.936408 8149 tablet_bootstrap.cc:492] T fbe595e147b4459ab774f1129b6edc46 P 65c308b30fa54aaba7bd5203d9b258a1: Bootstrap starting.
I20260621 07:47:40.938007 8151 tablet_bootstrap.cc:492] T fbe595e147b4459ab774f1129b6edc46 P efd8a37d6ac44bb4b4ef2f0e61cdb11c: Bootstrap starting.
I20260621 07:47:40.938817 8149 tablet_bootstrap.cc:654] T fbe595e147b4459ab774f1129b6edc46 P 65c308b30fa54aaba7bd5203d9b258a1: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:40.939566 8149 log.cc:824] T fbe595e147b4459ab774f1129b6edc46 P 65c308b30fa54aaba7bd5203d9b258a1: Log is configured to *not* fsync() on all Append() calls
I20260621 07:47:40.940049 8151 tablet_bootstrap.cc:654] T fbe595e147b4459ab774f1129b6edc46 P efd8a37d6ac44bb4b4ef2f0e61cdb11c: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:40.940227 8150 tablet_bootstrap.cc:492] T fbe595e147b4459ab774f1129b6edc46 P eecd298bb09548e0b4f4edffda7cbe91: Bootstrap starting.
I20260621 07:47:40.940841 8151 log.cc:824] T fbe595e147b4459ab774f1129b6edc46 P efd8a37d6ac44bb4b4ef2f0e61cdb11c: Log is configured to *not* fsync() on all Append() calls
I20260621 07:47:40.941594 8149 tablet_bootstrap.cc:492] T fbe595e147b4459ab774f1129b6edc46 P 65c308b30fa54aaba7bd5203d9b258a1: No bootstrap required, opened a new log
I20260621 07:47:40.941781 8149 ts_tablet_manager.cc:1397] T fbe595e147b4459ab774f1129b6edc46 P 65c308b30fa54aaba7bd5203d9b258a1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260621 07:47:40.942589 8150 tablet_bootstrap.cc:654] T fbe595e147b4459ab774f1129b6edc46 P eecd298bb09548e0b4f4edffda7cbe91: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:40.943058 8151 tablet_bootstrap.cc:492] T fbe595e147b4459ab774f1129b6edc46 P efd8a37d6ac44bb4b4ef2f0e61cdb11c: No bootstrap required, opened a new log
I20260621 07:47:40.943231 8151 ts_tablet_manager.cc:1397] T fbe595e147b4459ab774f1129b6edc46 P efd8a37d6ac44bb4b4ef2f0e61cdb11c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260621 07:47:40.943271 8150 log.cc:824] T fbe595e147b4459ab774f1129b6edc46 P eecd298bb09548e0b4f4edffda7cbe91: Log is configured to *not* fsync() on all Append() calls
I20260621 07:47:40.944950 8150 tablet_bootstrap.cc:492] T fbe595e147b4459ab774f1129b6edc46 P eecd298bb09548e0b4f4edffda7cbe91: No bootstrap required, opened a new log
I20260621 07:47:40.945120 8150 ts_tablet_manager.cc:1397] T fbe595e147b4459ab774f1129b6edc46 P eecd298bb09548e0b4f4edffda7cbe91: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260621 07:47:40.945240 8149 raft_consensus.cc:348] T fbe595e147b4459ab774f1129b6edc46 P 65c308b30fa54aaba7bd5203d9b258a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65c308b30fa54aaba7bd5203d9b258a1" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40815 } } peers { permanent_uuid: "eecd298bb09548e0b4f4edffda7cbe91" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43223 } } peers { permanent_uuid: "efd8a37d6ac44bb4b4ef2f0e61cdb11c" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45293 } }
I20260621 07:47:40.945436 8149 raft_consensus.cc:374] T fbe595e147b4459ab774f1129b6edc46 P 65c308b30fa54aaba7bd5203d9b258a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:40.945487 8149 raft_consensus.cc:729] T fbe595e147b4459ab774f1129b6edc46 P 65c308b30fa54aaba7bd5203d9b258a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c308b30fa54aaba7bd5203d9b258a1, State: Initialized, Role: FOLLOWER
I20260621 07:47:40.945854 8149 consensus_queue.cc:260] T fbe595e147b4459ab774f1129b6edc46 P 65c308b30fa54aaba7bd5203d9b258a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65c308b30fa54aaba7bd5203d9b258a1" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40815 } } peers { permanent_uuid: "eecd298bb09548e0b4f4edffda7cbe91" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43223 } } peers { permanent_uuid: "efd8a37d6ac44bb4b4ef2f0e61cdb11c" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45293 } }
I20260621 07:47:40.946681 8149 ts_tablet_manager.cc:1428] T fbe595e147b4459ab774f1129b6edc46 P 65c308b30fa54aaba7bd5203d9b258a1: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260621 07:47:40.947073 8151 raft_consensus.cc:348] T fbe595e147b4459ab774f1129b6edc46 P efd8a37d6ac44bb4b4ef2f0e61cdb11c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65c308b30fa54aaba7bd5203d9b258a1" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40815 } } peers { permanent_uuid: "eecd298bb09548e0b4f4edffda7cbe91" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43223 } } peers { permanent_uuid: "efd8a37d6ac44bb4b4ef2f0e61cdb11c" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45293 } }
I20260621 07:47:40.947301 8151 raft_consensus.cc:374] T fbe595e147b4459ab774f1129b6edc46 P efd8a37d6ac44bb4b4ef2f0e61cdb11c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:40.947357 8151 raft_consensus.cc:729] T fbe595e147b4459ab774f1129b6edc46 P efd8a37d6ac44bb4b4ef2f0e61cdb11c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efd8a37d6ac44bb4b4ef2f0e61cdb11c, State: Initialized, Role: FOLLOWER
I20260621 07:47:40.947674 7980 heartbeater.cc:503] Master 127.0.192.254:40701 was elected leader, sending a full tablet report...
I20260621 07:47:40.947757 8151 consensus_queue.cc:260] T fbe595e147b4459ab774f1129b6edc46 P efd8a37d6ac44bb4b4ef2f0e61cdb11c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65c308b30fa54aaba7bd5203d9b258a1" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40815 } } peers { permanent_uuid: "eecd298bb09548e0b4f4edffda7cbe91" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43223 } } peers { permanent_uuid: "efd8a37d6ac44bb4b4ef2f0e61cdb11c" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45293 } }
I20260621 07:47:40.948776 8151 ts_tablet_manager.cc:1428] T fbe595e147b4459ab774f1129b6edc46 P efd8a37d6ac44bb4b4ef2f0e61cdb11c: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260621 07:47:40.949240 8150 raft_consensus.cc:348] T fbe595e147b4459ab774f1129b6edc46 P eecd298bb09548e0b4f4edffda7cbe91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65c308b30fa54aaba7bd5203d9b258a1" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40815 } } peers { permanent_uuid: "eecd298bb09548e0b4f4edffda7cbe91" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43223 } } peers { permanent_uuid: "efd8a37d6ac44bb4b4ef2f0e61cdb11c" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45293 } }
I20260621 07:47:40.949461 8150 raft_consensus.cc:374] T fbe595e147b4459ab774f1129b6edc46 P eecd298bb09548e0b4f4edffda7cbe91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:40.949513 8150 raft_consensus.cc:729] T fbe595e147b4459ab774f1129b6edc46 P eecd298bb09548e0b4f4edffda7cbe91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eecd298bb09548e0b4f4edffda7cbe91, State: Initialized, Role: FOLLOWER
I20260621 07:47:40.949909 8150 consensus_queue.cc:260] T fbe595e147b4459ab774f1129b6edc46 P eecd298bb09548e0b4f4edffda7cbe91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65c308b30fa54aaba7bd5203d9b258a1" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40815 } } peers { permanent_uuid: "eecd298bb09548e0b4f4edffda7cbe91" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43223 } } peers { permanent_uuid: "efd8a37d6ac44bb4b4ef2f0e61cdb11c" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45293 } }
I20260621 07:47:40.950629 8116 heartbeater.cc:503] Master 127.0.192.254:40701 was elected leader, sending a full tablet report...
I20260621 07:47:40.950815 7843 heartbeater.cc:503] Master 127.0.192.254:40701 was elected leader, sending a full tablet report...
I20260621 07:47:40.950894 8150 ts_tablet_manager.cc:1428] T fbe595e147b4459ab774f1129b6edc46 P eecd298bb09548e0b4f4edffda7cbe91: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260621 07:47:40.958595 7982 tablet.cc:2369] T fbe595e147b4459ab774f1129b6edc46 P 65c308b30fa54aaba7bd5203d9b258a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 07:47:41.031682 7845 tablet.cc:2369] T fbe595e147b4459ab774f1129b6edc46 P efd8a37d6ac44bb4b4ef2f0e61cdb11c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:47:41.079196 8155 raft_consensus.cc:482] T fbe595e147b4459ab774f1129b6edc46 P 65c308b30fa54aaba7bd5203d9b258a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:47:41.079357 8155 raft_consensus.cc:504] T fbe595e147b4459ab774f1129b6edc46 P 65c308b30fa54aaba7bd5203d9b258a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65c308b30fa54aaba7bd5203d9b258a1" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40815 } } peers { permanent_uuid: "eecd298bb09548e0b4f4edffda7cbe91" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43223 } } peers { permanent_uuid: "efd8a37d6ac44bb4b4ef2f0e61cdb11c" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45293 } }
I20260621 07:47:41.080314 8155 leader_election.cc:288] T fbe595e147b4459ab774f1129b6edc46 P 65c308b30fa54aaba7bd5203d9b258a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eecd298bb09548e0b4f4edffda7cbe91 (127.0.192.195:43223), efd8a37d6ac44bb4b4ef2f0e61cdb11c (127.0.192.193:45293)
I20260621 07:47:41.084025 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:41.080619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:47257 (local address 127.0.192.195:43223)
0621 07:47:41.080777 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:41.080781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:41.080795 (+ 14us) server_negotiation.cc:407] Connection header received
0621 07:47:41.080907 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:41.080910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:41.080963 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:41.081053 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:41.081378 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:41.081949 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:41.083030 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:41.083534 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:41.083614 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:41.083682 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:41.083852 (+ 170us) server_negotiation.cc:299] Negotiation successful
0621 07:47:41.083918 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260621 07:47:41.084021 7984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:41.080538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.195:43223 (local address 127.0.192.194:47257)
0621 07:47:41.080700 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:41.080717 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:41.080797 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:41.081074 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:41.081077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:41.081093 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:41.081243 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:41.081253 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:41.082081 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:41.082084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:41.082924 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:41.082933 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:41.083639 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:41.083644 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:41.083726 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:41.083755 (+ 29us) client_negotiation.cc:769] Sending connection context
0621 07:47:41.083817 (+ 62us) client_negotiation.cc:240] Negotiation successful
0621 07:47:41.083888 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"mutex_wait_us":37}
I20260621 07:47:41.084056 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:41.080538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:45293 (local address 127.0.192.194:49791)
0621 07:47:41.080870 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:41.080884 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:41.080975 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:41.081279 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:41.081281 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:41.081298 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:41.081407 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:41.081413 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:41.082233 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:41.082235 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:41.083117 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:41.083126 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:41.083695 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:41.083700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:41.083783 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:41.083797 (+ 14us) client_negotiation.cc:769] Sending connection context
0621 07:47:41.083881 (+ 84us) client_negotiation.cc:240] Negotiation successful
0621 07:47:41.083937 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":240}
I20260621 07:47:41.084069 8148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:41.080609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:49791 (local address 127.0.192.193:45293)
0621 07:47:41.080737 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:41.080740 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:41.080959 (+ 219us) server_negotiation.cc:407] Connection header received
0621 07:47:41.081102 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:41.081105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:41.081194 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:41.081266 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:41.081512 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:41.082141 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:41.083220 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:41.083567 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:41.083683 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:41.083729 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:41.083911 (+ 182us) server_negotiation.cc:299] Negotiation successful
0621 07:47:41.083971 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260621 07:47:41.084525 8069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbe595e147b4459ab774f1129b6edc46" candidate_uuid: "65c308b30fa54aaba7bd5203d9b258a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eecd298bb09548e0b4f4edffda7cbe91" is_pre_election: true
I20260621 07:47:41.084518 7796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbe595e147b4459ab774f1129b6edc46" candidate_uuid: "65c308b30fa54aaba7bd5203d9b258a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efd8a37d6ac44bb4b4ef2f0e61cdb11c" is_pre_election: true
I20260621 07:47:41.084779 8069 raft_consensus.cc:2454] T fbe595e147b4459ab774f1129b6edc46 P eecd298bb09548e0b4f4edffda7cbe91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c308b30fa54aaba7bd5203d9b258a1 in term 0.
I20260621 07:47:41.084779 7796 raft_consensus.cc:2454] T fbe595e147b4459ab774f1129b6edc46 P efd8a37d6ac44bb4b4ef2f0e61cdb11c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c308b30fa54aaba7bd5203d9b258a1 in term 0.
I20260621 07:47:41.085130 7869 leader_election.cc:302] T fbe595e147b4459ab774f1129b6edc46 P 65c308b30fa54aaba7bd5203d9b258a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c308b30fa54aaba7bd5203d9b258a1, efd8a37d6ac44bb4b4ef2f0e61cdb11c; no voters:
I20260621 07:47:41.085332 8155 raft_consensus.cc:2781] T fbe595e147b4459ab774f1129b6edc46 P 65c308b30fa54aaba7bd5203d9b258a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:47:41.085400 8155 raft_consensus.cc:482] T fbe595e147b4459ab774f1129b6edc46 P 65c308b30fa54aaba7bd5203d9b258a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:47:41.085436 8155 raft_consensus.cc:3037] T fbe595e147b4459ab774f1129b6edc46 P 65c308b30fa54aaba7bd5203d9b258a1 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:41.086300 8155 raft_consensus.cc:504] T fbe595e147b4459ab774f1129b6edc46 P 65c308b30fa54aaba7bd5203d9b258a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65c308b30fa54aaba7bd5203d9b258a1" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40815 } } peers { permanent_uuid: "eecd298bb09548e0b4f4edffda7cbe91" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43223 } } peers { permanent_uuid: "efd8a37d6ac44bb4b4ef2f0e61cdb11c" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45293 } }
I20260621 07:47:41.086557 8155 leader_election.cc:288] T fbe595e147b4459ab774f1129b6edc46 P 65c308b30fa54aaba7bd5203d9b258a1 [CANDIDATE]: Term 1 election: Requested vote from peers eecd298bb09548e0b4f4edffda7cbe91 (127.0.192.195:43223), efd8a37d6ac44bb4b4ef2f0e61cdb11c (127.0.192.193:45293)
I20260621 07:47:41.086815 7796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbe595e147b4459ab774f1129b6edc46" candidate_uuid: "65c308b30fa54aaba7bd5203d9b258a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efd8a37d6ac44bb4b4ef2f0e61cdb11c"
I20260621 07:47:41.086815 8069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbe595e147b4459ab774f1129b6edc46" candidate_uuid: "65c308b30fa54aaba7bd5203d9b258a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eecd298bb09548e0b4f4edffda7cbe91"
I20260621 07:47:41.086916 8069 raft_consensus.cc:3037] T fbe595e147b4459ab774f1129b6edc46 P eecd298bb09548e0b4f4edffda7cbe91 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:41.086916 7796 raft_consensus.cc:3037] T fbe595e147b4459ab774f1129b6edc46 P efd8a37d6ac44bb4b4ef2f0e61cdb11c [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:41.088083 8069 raft_consensus.cc:2454] T fbe595e147b4459ab774f1129b6edc46 P eecd298bb09548e0b4f4edffda7cbe91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c308b30fa54aaba7bd5203d9b258a1 in term 1.
I20260621 07:47:41.088083 7796 raft_consensus.cc:2454] T fbe595e147b4459ab774f1129b6edc46 P efd8a37d6ac44bb4b4ef2f0e61cdb11c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c308b30fa54aaba7bd5203d9b258a1 in term 1.
I20260621 07:47:41.088339 7868 leader_election.cc:302] T fbe595e147b4459ab774f1129b6edc46 P 65c308b30fa54aaba7bd5203d9b258a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c308b30fa54aaba7bd5203d9b258a1, eecd298bb09548e0b4f4edffda7cbe91; no voters:
I20260621 07:47:41.088521 8155 raft_consensus.cc:2781] T fbe595e147b4459ab774f1129b6edc46 P 65c308b30fa54aaba7bd5203d9b258a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:47:41.088785 8155 raft_consensus.cc:686] T fbe595e147b4459ab774f1129b6edc46 P 65c308b30fa54aaba7bd5203d9b258a1 [term 1 LEADER]: Becoming Leader. State: Replica: 65c308b30fa54aaba7bd5203d9b258a1, State: Running, Role: LEADER
I20260621 07:47:41.089105 8155 consensus_queue.cc:237] T fbe595e147b4459ab774f1129b6edc46 P 65c308b30fa54aaba7bd5203d9b258a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65c308b30fa54aaba7bd5203d9b258a1" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40815 } } peers { permanent_uuid: "eecd298bb09548e0b4f4edffda7cbe91" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43223 } } peers { permanent_uuid: "efd8a37d6ac44bb4b4ef2f0e61cdb11c" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45293 } }
I20260621 07:47:41.091904 6870 catalog_manager.cc:5495] T fbe595e147b4459ab774f1129b6edc46 P 65c308b30fa54aaba7bd5203d9b258a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65c308b30fa54aaba7bd5203d9b258a1 (127.0.192.194). New cstate: current_term: 1 leader_uuid: "65c308b30fa54aaba7bd5203d9b258a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65c308b30fa54aaba7bd5203d9b258a1" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eecd298bb09548e0b4f4edffda7cbe91" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efd8a37d6ac44bb4b4ef2f0e61cdb11c" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45293 } health_report { overall_health: UNKNOWN } } }
I20260621 07:47:41.113404 8144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:41.110722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41028 (local address 127.0.192.194:40815)
0621 07:47:41.110814 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:41.110817 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:41.110827 (+ 10us) server_negotiation.cc:407] Connection header received
0621 07:47:41.110964 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:41.110966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:41.111009 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:41.111086 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:47:41.111320 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:41.111710 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:41.112377 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:41.112548 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:41.112666 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:41.112860 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:47:41.113168 (+ 308us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:47:41.113244 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:41.113294 (+ 50us) server_negotiation.cc:299] Negotiation successful
0621 07:47:41.113330 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260621 07:47:41.118705 7796 raft_consensus.cc:1261] T fbe595e147b4459ab774f1129b6edc46 P efd8a37d6ac44bb4b4ef2f0e61cdb11c [term 1 FOLLOWER]: Refusing update from remote peer 65c308b30fa54aaba7bd5203d9b258a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:47:41.118718 8069 raft_consensus.cc:1261] T fbe595e147b4459ab774f1129b6edc46 P eecd298bb09548e0b4f4edffda7cbe91 [term 1 FOLLOWER]: Refusing update from remote peer 65c308b30fa54aaba7bd5203d9b258a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:47:41.119266 8155 consensus_queue.cc:1035] T fbe595e147b4459ab774f1129b6edc46 P 65c308b30fa54aaba7bd5203d9b258a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efd8a37d6ac44bb4b4ef2f0e61cdb11c" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:47:41.119406 8158 consensus_queue.cc:1035] T fbe595e147b4459ab774f1129b6edc46 P 65c308b30fa54aaba7bd5203d9b258a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eecd298bb09548e0b4f4edffda7cbe91" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260621 07:47:41.127218 8118 tablet.cc:2369] T fbe595e147b4459ab774f1129b6edc46 P eecd298bb09548e0b4f4edffda7cbe91: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:47:41.129451 8162 mvcc.cc:205] Tried to move back new op lower bound from 7299186938335703040 to 7299186938223095808. Current Snapshot: MvccSnapshot[applied={T|T < 7299186938335703040}]
I20260621 07:47:41.130621 8163 mvcc.cc:205] Tried to move back new op lower bound from 7299186938335703040 to 7299186938223095808. Current Snapshot: MvccSnapshot[applied={T|T < 7299186938335703040 or (T in {7299186938335703040})}]
I20260621 07:47:41.132596 6870 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:47:41.132726 6870 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:47:41.137313 6870 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8072ff3594143ca8e36eb920eae693c: Sending DeleteTablet for 3 replicas of tablet fbe595e147b4459ab774f1129b6edc46
I20260621 07:47:41.137899 7913 tablet_service.cc:1514] Processing DeleteTablet for tablet fbe595e147b4459ab774f1129b6edc46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:47:41 UTC) from {username='oryx', principal='oryx/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:41024
I20260621 07:47:41.138198 8049 tablet_service.cc:1514] Processing DeleteTablet for tablet fbe595e147b4459ab774f1129b6edc46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:47:41 UTC) from {username='oryx', principal='oryx/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:43310
I20260621 07:47:41.138607 8171 tablet_replica.cc:330] T fbe595e147b4459ab774f1129b6edc46 P eecd298bb09548e0b4f4edffda7cbe91: stopping tablet replica
I20260621 07:47:41.138864 8171 raft_consensus.cc:2229] T fbe595e147b4459ab774f1129b6edc46 P eecd298bb09548e0b4f4edffda7cbe91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:47:41.139078 8171 raft_consensus.cc:2258] T fbe595e147b4459ab774f1129b6edc46 P eecd298bb09548e0b4f4edffda7cbe91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:47:41.140221 8171 ts_tablet_manager.cc:1905] T fbe595e147b4459ab774f1129b6edc46 P eecd298bb09548e0b4f4edffda7cbe91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:47:41.141381 8170 tablet_replica.cc:330] T fbe595e147b4459ab774f1129b6edc46 P 65c308b30fa54aaba7bd5203d9b258a1: stopping tablet replica
I20260621 07:47:41.141664 8170 raft_consensus.cc:2229] T fbe595e147b4459ab774f1129b6edc46 P 65c308b30fa54aaba7bd5203d9b258a1 [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:47:41.141927 7776 tablet_service.cc:1514] Processing DeleteTablet for tablet fbe595e147b4459ab774f1129b6edc46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:47:41 UTC) from {username='oryx', principal='oryx/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:51748
I20260621 07:47:41.142095 8170 raft_consensus.cc:2258] T fbe595e147b4459ab774f1129b6edc46 P 65c308b30fa54aaba7bd5203d9b258a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:47:41.142763 8171 ts_tablet_manager.cc:1918] T fbe595e147b4459ab774f1129b6edc46 P eecd298bb09548e0b4f4edffda7cbe91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:47:41.142853 8171 log.cc:1196] T fbe595e147b4459ab774f1129b6edc46 P eecd298bb09548e0b4f4edffda7cbe91: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-2/wal/wals/fbe595e147b4459ab774f1129b6edc46
I20260621 07:47:41.143154 8171 ts_tablet_manager.cc:1939] T fbe595e147b4459ab774f1129b6edc46 P eecd298bb09548e0b4f4edffda7cbe91: Deleting consensus metadata
I20260621 07:47:41.143992 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 6839
I20260621 07:47:41.144352 8170 ts_tablet_manager.cc:1905] T fbe595e147b4459ab774f1129b6edc46 P 65c308b30fa54aaba7bd5203d9b258a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:47:41.146654 8170 ts_tablet_manager.cc:1918] T fbe595e147b4459ab774f1129b6edc46 P 65c308b30fa54aaba7bd5203d9b258a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:47:41.146734 8170 log.cc:1196] T fbe595e147b4459ab774f1129b6edc46 P 65c308b30fa54aaba7bd5203d9b258a1: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-1/wal/wals/fbe595e147b4459ab774f1129b6edc46
I20260621 07:47:41.146987 8170 ts_tablet_manager.cc:1939] T fbe595e147b4459ab774f1129b6edc46 P 65c308b30fa54aaba7bd5203d9b258a1: Deleting consensus metadata
I20260621 07:47:41.148717 8173 tablet_replica.cc:330] T fbe595e147b4459ab774f1129b6edc46 P efd8a37d6ac44bb4b4ef2f0e61cdb11c: stopping tablet replica
I20260621 07:47:41.148952 8173 raft_consensus.cc:2229] T fbe595e147b4459ab774f1129b6edc46 P efd8a37d6ac44bb4b4ef2f0e61cdb11c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:47:41.149178 8173 raft_consensus.cc:2258] T fbe595e147b4459ab774f1129b6edc46 P efd8a37d6ac44bb4b4ef2f0e61cdb11c [term 1 FOLLOWER]: Raft consensus is shut down!
W20260621 07:47:41.150184 8004 connection.cc:535] server connection from 127.0.0.1:43310 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260621 07:47:41.150458 7868 connection.cc:535] server connection from 127.0.0.1:41024 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260621 07:47:41.151038 8173 ts_tablet_manager.cc:1905] T fbe595e147b4459ab774f1129b6edc46 P efd8a37d6ac44bb4b4ef2f0e61cdb11c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:47:41.153439 8173 ts_tablet_manager.cc:1918] T fbe595e147b4459ab774f1129b6edc46 P efd8a37d6ac44bb4b4ef2f0e61cdb11c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:47:41.153522 8173 log.cc:1196] T fbe595e147b4459ab774f1129b6edc46 P efd8a37d6ac44bb4b4ef2f0e61cdb11c: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-0/wal/wals/fbe595e147b4459ab774f1129b6edc46
I20260621 07:47:41.153779 8173 ts_tablet_manager.cc:1939] T fbe595e147b4459ab774f1129b6edc46 P efd8a37d6ac44bb4b4ef2f0e61cdb11c: Deleting consensus metadata
W20260621 07:47:41.154580 7732 connection.cc:441] server connection from 127.0.0.1:51748 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51748 (request call id 1) could send its response
I20260621 07:47:41.161904 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:41.152708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53886 (local address 127.0.192.252:36421)
0621 07:47:41.152786 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:41.152790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:41.153423 (+ 633us) server_negotiation.cc:407] Connection header received
0621 07:47:41.153616 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:41.153619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:41.153654 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:41.153693 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:41.154600 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:41.155433 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:41.156516 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:41.156841 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:41.156935 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:41.158726 (+ 1791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:41.158730 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:41.158732 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:41.158753 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:41.160265 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:41.160766 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:41.160770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:41.160771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:41.160845 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:41.161097 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:41.161100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:41.161101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:41.161266 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:41.161373 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:41.161685 (+ 312us) server_negotiation.cc:299] Negotiation successful
0621 07:47:41.161779 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260621 07:47:41.167292 8127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:41.143319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40254 (local address 127.0.192.253:43105)
0621 07:47:41.143447 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:41.143451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:41.159679 (+ 16228us) server_negotiation.cc:407] Connection header received
0621 07:47:41.159786 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:41.159789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:41.159834 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:41.159900 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:41.160668 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:41.161330 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:41.162321 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:41.162625 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:41.162688 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:41.164005 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:41.164009 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:41.164011 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:41.164038 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:41.165670 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:41.166115 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:41.166121 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:41.166124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:41.166196 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:41.166451 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:41.166454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:41.166456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:41.166606 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:41.166724 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:41.166935 (+ 211us) server_negotiation.cc:299] Negotiation successful
0621 07:47:41.167146 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43,"spinlock_wait_cycles":4992}
I20260621 07:47:41.167709 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:41.159985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40266 (local address 127.0.192.253:43105)
0621 07:47:41.160078 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:41.160082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:41.160371 (+ 289us) server_negotiation.cc:407] Connection header received
0621 07:47:41.160542 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:41.160545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:41.160605 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:41.160653 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:41.161289 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:41.161960 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:41.163120 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:41.163402 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:41.163499 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:41.164856 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:41.164860 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:41.164862 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:41.164883 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:41.166162 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:41.166561 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:41.166564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:41.166566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:41.166642 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:41.166903 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:41.166906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:41.166908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:41.167071 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:41.167155 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:41.167426 (+ 271us) server_negotiation.cc:299] Negotiation successful
0621 07:47:41.167545 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33,"spinlock_wait_cycles":2816}
I20260621 07:47:41.174425 8130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:41.142802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53870 (local address 127.0.192.252:36421)
0621 07:47:41.142923 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:41.142929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:41.167151 (+ 24222us) server_negotiation.cc:407] Connection header received
0621 07:47:41.167278 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:41.167280 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:41.167321 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:41.167371 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:41.168114 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:41.168898 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:41.169860 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:41.170042 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:41.170129 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:41.171414 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:41.171417 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:41.171418 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:41.171437 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:41.172893 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:41.173321 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:41.173324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:41.173325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:41.173393 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:41.173654 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:41.173659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:41.173661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:41.173823 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:41.173902 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:41.174178 (+ 276us) server_negotiation.cc:299] Negotiation successful
0621 07:47:41.174261 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260621 07:47:42.128332 8180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:42.127938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:40701 (local address 127.0.192.194:38529)
0621 07:47:42.128163 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:42.128240 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":66,"threads_started":1}
W20260621 07:47:42.128671 7980 heartbeater.cc:644] Failed to heartbeat to 127.0.192.254:40701 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.192.254:40701: connect: Connection refused (error 111)
I20260621 07:47:42.130406 8181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:42.129992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:40701 (local address 127.0.192.195:36501)
0621 07:47:42.130220 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:42.130297 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":77,"threads_started":1}
W20260621 07:47:42.130767 8116 heartbeater.cc:644] Failed to heartbeat to 127.0.192.254:40701 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.192.254:40701: connect: Connection refused (error 111)
I20260621 07:47:42.133775 8182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:42.133466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:40701 (local address 127.0.192.193:46351)
0621 07:47:42.133651 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:42.133709 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":48,"threads_started":1}
W20260621 07:47:42.133983 7843 heartbeater.cc:644] Failed to heartbeat to 127.0.192.254:40701 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.192.254:40701: connect: Connection refused (error 111)
I20260621 07:47:42.637655 8184 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c8072ff3594143ca8e36eb920eae693c)
I20260621 07:47:42.637656 8183 raft_consensus.cc:482] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c8072ff3594143ca8e36eb920eae693c)
I20260621 07:47:42.637868 8183 raft_consensus.cc:504] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: false } } peers { permanent_uuid: "301f9ea85622455aaa68534f71280c4d" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 36421 } attrs { promote: false } }
I20260621 07:47:42.637880 8184 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: false } } peers { permanent_uuid: "301f9ea85622455aaa68534f71280c4d" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 36421 } attrs { promote: false } }
I20260621 07:47:42.638816 8184 leader_election.cc:288] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c8072ff3594143ca8e36eb920eae693c (127.0.192.254:40701), 301f9ea85622455aaa68534f71280c4d (127.0.192.252:36421)
I20260621 07:47:42.639137 8183 leader_election.cc:288] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c8072ff3594143ca8e36eb920eae693c (127.0.192.254:40701), 2b7be1241e6d426d87e026a0d4b8292c (127.0.192.253:43105)
I20260621 07:47:42.639426 8185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:42.638935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:40701 (local address 127.0.0.1:34640)
0621 07:47:42.639256 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:42.639325 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":68,"threads_started":1}
I20260621 07:47:42.639478 8187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:42.638938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:40701 (local address 127.0.0.1:34650)
0621 07:47:42.639324 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:42.639388 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":65,"threads_started":1}
W20260621 07:47:42.639576 7458 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.192.254:40701: connect: Connection refused (error 111)
W20260621 07:47:42.639735 7650 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.192.254:40701: connect: Connection refused (error 111)
I20260621 07:47:42.640480 8185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:42.640204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:40701 (local address 127.0.0.1:34660)
0621 07:47:42.640352 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:42.640393 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260621 07:47:42.640605 8186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:42.640387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:40701 (local address 127.0.0.1:34666)
0621 07:47:42.640483 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:42.640532 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260621 07:47:42.640614 7458 leader_election.cc:334] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8072ff3594143ca8e36eb920eae693c (127.0.192.254:40701): Network error: Client connection negotiation failed: client connection to 127.0.192.254:40701: connect: Connection refused (error 111)
W20260621 07:47:42.640739 7650 leader_election.cc:334] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8072ff3594143ca8e36eb920eae693c (127.0.192.254:40701): Network error: Client connection negotiation failed: client connection to 127.0.192.254:40701: connect: Connection refused (error 111)
I20260621 07:47:42.642838 8189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:42.639264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:36421 (local address 127.0.0.1:53902)
0621 07:47:42.639510 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:42.639524 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:42.639616 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:42.640011 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:42.640014 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:42.640029 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:42.640143 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:42.640149 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:42.641110 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:42.641114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:42.641801 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:42.641808 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:42.642436 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:42.642439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:42.642562 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:42.642579 (+ 17us) client_negotiation.cc:769] Sending connection context
0621 07:47:42.642658 (+ 79us) client_negotiation.cc:240] Negotiation successful
0621 07:47:42.642729 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":43,"thread_start_us":50,"threads_started":1}
I20260621 07:47:42.642839 8190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:42.639572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53902 (local address 127.0.192.252:36421)
0621 07:47:42.639820 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:42.639824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:42.639835 (+ 11us) server_negotiation.cc:407] Connection header received
0621 07:47:42.639870 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:42.639875 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:42.639917 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:42.640243 (+ 326us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:42.640328 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:42.641014 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:42.641923 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:42.642322 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:42.642425 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:42.642478 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:42.642672 (+ 194us) server_negotiation.cc:299] Negotiation successful
0621 07:47:42.642735 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":47,"threads_started":1}
I20260621 07:47:42.643172 8187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:42.638970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.253:43105 (local address 127.0.0.1:40272)
0621 07:47:42.639745 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:42.639757 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:42.639854 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:42.640135 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:42.640138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:42.640152 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:42.640247 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:42.640253 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:42.641436 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:42.641439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:42.642144 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:42.642153 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:42.642775 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:42.642779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:42.642873 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:42.642914 (+ 41us) client_negotiation.cc:769] Sending connection context
0621 07:47:42.642993 (+ 79us) client_negotiation.cc:240] Negotiation successful
0621 07:47:42.643057 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"thread_start_us":79,"threads_started":1}
I20260621 07:47:42.643226 7673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2b7be1241e6d426d87e026a0d4b8292c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "301f9ea85622455aaa68534f71280c4d" is_pre_election: true
I20260621 07:47:42.643448 7673 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b7be1241e6d426d87e026a0d4b8292c in term 1.
I20260621 07:47:42.643759 7457 leader_election.cc:302] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2b7be1241e6d426d87e026a0d4b8292c, 301f9ea85622455aaa68534f71280c4d; no voters: c8072ff3594143ca8e36eb920eae693c
I20260621 07:47:42.643877 8188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:42.639131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40272 (local address 127.0.192.253:43105)
0621 07:47:42.639375 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:42.639378 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:42.639921 (+ 543us) server_negotiation.cc:407] Connection header received
0621 07:47:42.639964 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:42.639968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:42.640011 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:42.640070 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:42.640345 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:42.641314 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:42.642269 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:42.642642 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:42.642765 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:42.642812 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:42.643093 (+ 281us) server_negotiation.cc:299] Negotiation successful
0621 07:47:42.643704 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":198,"thread_start_us":41,"threads_started":1}
I20260621 07:47:42.644006 8184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260621 07:47:42.644060 8184 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [term 1 FOLLOWER]: Starting leader election (detected failure of leader c8072ff3594143ca8e36eb920eae693c)
I20260621 07:47:42.644097 8184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [term 1 FOLLOWER]: Advancing to term 2
I20260621 07:47:42.644208 7481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "301f9ea85622455aaa68534f71280c4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2b7be1241e6d426d87e026a0d4b8292c" is_pre_election: true
I20260621 07:47:42.645151 8184 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: false } } peers { permanent_uuid: "301f9ea85622455aaa68534f71280c4d" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 36421 } attrs { promote: false } }
I20260621 07:47:42.645419 7481 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 301f9ea85622455aaa68534f71280c4d in current term 2: Already voted for candidate 2b7be1241e6d426d87e026a0d4b8292c in this term.
I20260621 07:47:42.645502 8184 leader_election.cc:288] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [CANDIDATE]: Term 2 election: Requested vote from peers c8072ff3594143ca8e36eb920eae693c (127.0.192.254:40701), 301f9ea85622455aaa68534f71280c4d (127.0.192.252:36421)
I20260621 07:47:42.645777 8189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:42.645574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:40701 (local address 127.0.0.1:34668)
0621 07:47:42.645687 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:42.645728 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260621 07:47:42.645915 7673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2b7be1241e6d426d87e026a0d4b8292c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "301f9ea85622455aaa68534f71280c4d"
I20260621 07:47:42.646031 7673 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [term 1 FOLLOWER]: Advancing to term 2
I20260621 07:47:42.646108 7647 leader_election.cc:302] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 301f9ea85622455aaa68534f71280c4d; no voters: 2b7be1241e6d426d87e026a0d4b8292c, c8072ff3594143ca8e36eb920eae693c
I20260621 07:47:42.646384 8189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:42.646145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:40701 (local address 127.0.0.1:34682)
0621 07:47:42.646276 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:42.646330 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260621 07:47:42.646509 7458 leader_election.cc:334] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c8072ff3594143ca8e36eb920eae693c (127.0.192.254:40701): Network error: Client connection negotiation failed: client connection to 127.0.192.254:40701: connect: Connection refused (error 111)
I20260621 07:47:42.646999 7673 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b7be1241e6d426d87e026a0d4b8292c in term 2.
I20260621 07:47:42.647104 8183 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 07:47:42.647274 7457 leader_election.cc:302] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2b7be1241e6d426d87e026a0d4b8292c, 301f9ea85622455aaa68534f71280c4d; no voters: c8072ff3594143ca8e36eb920eae693c
I20260621 07:47:42.647403 8184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [term 2 FOLLOWER]: Leader election won for term 2
I20260621 07:47:42.647560 8184 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [term 2 LEADER]: Becoming Leader. State: Replica: 2b7be1241e6d426d87e026a0d4b8292c, State: Running, Role: LEADER
I20260621 07:47:42.647804 8184 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: false } } peers { permanent_uuid: "301f9ea85622455aaa68534f71280c4d" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 36421 } attrs { promote: false } }
I20260621 07:47:42.648759 8193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b7be1241e6d426d87e026a0d4b8292c. Latest consensus state: current_term: 2 leader_uuid: "2b7be1241e6d426d87e026a0d4b8292c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: false } } peers { permanent_uuid: "301f9ea85622455aaa68534f71280c4d" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 36421 } attrs { promote: false } } }
I20260621 07:47:42.648842 8193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:42.649148 8195 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:47:42.651332 8195 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:47:42.651737 8195 catalog_manager.cc:1259] Loaded cluster ID: 2feba99244ef42a5b71acfd2c5120240
I20260621 07:47:42.651778 8195 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:47:42.652338 8195 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:47:42.652767 8195 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c: Loaded TSK: 0
I20260621 07:47:42.653406 8195 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:47:42.667140 7471 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40266:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:47:42.670009 7471 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:47:42.672531 8189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:42.672328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:40701 (local address 127.0.0.1:34694)
0621 07:47:42.672429 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:42.672479 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260621 07:47:42.672601 7673 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [term 2 FOLLOWER]: Refusing update from remote peer 2b7be1241e6d426d87e026a0d4b8292c: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260621 07:47:42.673101 8184 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [LEADER]: Connected to new peer: Peer: permanent_uuid: "301f9ea85622455aaa68534f71280c4d" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 36421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260621 07:47:42.673220 8189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:42.672999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:40701 (local address 127.0.0.1:34698)
0621 07:47:42.673126 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:42.673174 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260621 07:47:42.673328 7458 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c -> Peer c8072ff3594143ca8e36eb920eae693c (127.0.192.254:40701): Couldn't send request to peer c8072ff3594143ca8e36eb920eae693c. Status: Network error: Client connection negotiation failed: client connection to 127.0.192.254:40701: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260621 07:47:42.674268 8183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b7be1241e6d426d87e026a0d4b8292c. Latest consensus state: current_term: 2 leader_uuid: "2b7be1241e6d426d87e026a0d4b8292c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: false } } peers { permanent_uuid: "301f9ea85622455aaa68534f71280c4d" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 36421 } attrs { promote: false } } }
I20260621 07:47:42.674384 8183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:47:42.675256 8184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "2b7be1241e6d426d87e026a0d4b8292c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: false } } peers { permanent_uuid: "301f9ea85622455aaa68534f71280c4d" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 36421 } attrs { promote: false } } }
I20260621 07:47:42.675370 8184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:42.675511 8197 mvcc.cc:205] Tried to move back new op lower bound from 7299186944702484480 to 7299186944607203328. Current Snapshot: MvccSnapshot[applied={T|T < 7299186944702484480}]
I20260621 07:47:42.675686 8183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "2b7be1241e6d426d87e026a0d4b8292c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c8072ff3594143ca8e36eb920eae693c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 40701 } } peers { permanent_uuid: "2b7be1241e6d426d87e026a0d4b8292c" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 43105 } attrs { promote: false } } peers { permanent_uuid: "301f9ea85622455aaa68534f71280c4d" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 36421 } attrs { promote: false } } }
I20260621 07:47:42.675812 8183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 301f9ea85622455aaa68534f71280c4d [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:47:42.685289 8185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:42.681245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:45293 (local address 127.0.0.1:51754)
0621 07:47:42.681344 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:42.681361 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:42.681429 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:42.682198 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:42.682201 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:42.682217 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:42.682317 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:42.682322 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:42.683269 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:42.683271 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:42.684120 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:42.684128 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:42.684927 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:42.684930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:42.685004 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:42.685021 (+ 17us) client_negotiation.cc:769] Sending connection context
0621 07:47:42.685125 (+ 104us) client_negotiation.cc:240] Negotiation successful
0621 07:47:42.685183 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":66}
I20260621 07:47:42.685299 8204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:42.681596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51754 (local address 127.0.192.193:45293)
0621 07:47:42.681979 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:42.681983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:42.681995 (+ 12us) server_negotiation.cc:407] Connection header received
0621 07:47:42.682041 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:42.682044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:42.682091 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:42.682397 (+ 306us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:42.682508 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:42.683162 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:42.684250 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:42.684829 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:42.684929 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:42.684980 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:42.685140 (+ 160us) server_negotiation.cc:299] Negotiation successful
0621 07:47:42.685199 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":61,"threads_started":1}
I20260621 07:47:42.685561 8189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:42.681019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:40815 (local address 127.0.0.1:41036)
0621 07:47:42.681111 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:42.681123 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:42.681211 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:42.682206 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:42.682208 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:42.682242 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:42.682315 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:42.682322 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:42.683138 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:42.683143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:42.684408 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:42.684418 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:42.685233 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:42.685236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:42.685312 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:42.685330 (+ 18us) client_negotiation.cc:769] Sending connection context
0621 07:47:42.685371 (+ 41us) client_negotiation.cc:240] Negotiation successful
0621 07:47:42.685456 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":100,"spinlock_wait_cycles":384}
I20260621 07:47:42.685600 8202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:42.681486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.195:43223 (local address 127.0.0.1:43314)
0621 07:47:42.681685 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:42.681699 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:42.681779 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:42.682266 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:42.682268 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:42.682278 (+ 10us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:42.682368 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:42.682374 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:42.683254 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:42.683256 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:42.684411 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:42.684418 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:42.685321 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:42.685324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:42.685395 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:42.685410 (+ 15us) client_negotiation.cc:769] Sending connection context
0621 07:47:42.685462 (+ 52us) client_negotiation.cc:240] Negotiation successful
0621 07:47:42.685509 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":77,"thread_start_us":70,"threads_started":1}
I20260621 07:47:42.685933 7776 tablet_service.cc:1467] Processing CreateTablet for tablet 60f981df8beb4d9492ea4fe1c4f40fbd (DEFAULT_TABLE table=test-table [id=6a51804283a44561a61978f376783080]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:47:42.686050 8203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:42.681475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41036 (local address 127.0.192.194:40815)
0621 07:47:42.682001 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:42.682005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:42.682018 (+ 13us) server_negotiation.cc:407] Connection header received
0621 07:47:42.682057 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:42.682059 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:42.682096 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:42.682409 (+ 313us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:42.682571 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:42.683043 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:42.684540 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:42.685120 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:42.685792 (+ 672us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:42.685845 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:42.685876 (+ 31us) server_negotiation.cc:299] Negotiation successful
0621 07:47:42.685936 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":74,"threads_started":1}
I20260621 07:47:42.686204 7776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60f981df8beb4d9492ea4fe1c4f40fbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:42.686458 7913 tablet_service.cc:1467] Processing CreateTablet for tablet 60f981df8beb4d9492ea4fe1c4f40fbd (DEFAULT_TABLE table=test-table [id=6a51804283a44561a61978f376783080]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:47:42.686707 7913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60f981df8beb4d9492ea4fe1c4f40fbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:42.688529 8206 tablet_bootstrap.cc:492] T 60f981df8beb4d9492ea4fe1c4f40fbd P efd8a37d6ac44bb4b4ef2f0e61cdb11c: Bootstrap starting.
I20260621 07:47:42.688804 8205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:42.681765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43314 (local address 127.0.192.195:43223)
0621 07:47:42.681988 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:42.681992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:42.682005 (+ 13us) server_negotiation.cc:407] Connection header received
0621 07:47:42.682051 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:42.682054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:42.682095 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:42.682172 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:42.682469 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:42.683154 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:42.684540 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:42.685113 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:42.685227 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:42.685279 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:42.688565 (+ 3286us) server_negotiation.cc:299] Negotiation successful
0621 07:47:42.688676 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":76,"threads_started":1}
I20260621 07:47:42.689282 8049 tablet_service.cc:1467] Processing CreateTablet for tablet 60f981df8beb4d9492ea4fe1c4f40fbd (DEFAULT_TABLE table=test-table [id=6a51804283a44561a61978f376783080]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:47:42.689522 8049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60f981df8beb4d9492ea4fe1c4f40fbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:42.689563 8206 tablet_bootstrap.cc:654] T 60f981df8beb4d9492ea4fe1c4f40fbd P efd8a37d6ac44bb4b4ef2f0e61cdb11c: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:42.690081 8207 tablet_bootstrap.cc:492] T 60f981df8beb4d9492ea4fe1c4f40fbd P 65c308b30fa54aaba7bd5203d9b258a1: Bootstrap starting.
I20260621 07:47:42.690565 8206 tablet_bootstrap.cc:492] T 60f981df8beb4d9492ea4fe1c4f40fbd P efd8a37d6ac44bb4b4ef2f0e61cdb11c: No bootstrap required, opened a new log
I20260621 07:47:42.690644 8206 ts_tablet_manager.cc:1397] T 60f981df8beb4d9492ea4fe1c4f40fbd P efd8a37d6ac44bb4b4ef2f0e61cdb11c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:42.690879 8207 tablet_bootstrap.cc:654] T 60f981df8beb4d9492ea4fe1c4f40fbd P 65c308b30fa54aaba7bd5203d9b258a1: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:42.691007 8206 raft_consensus.cc:348] T 60f981df8beb4d9492ea4fe1c4f40fbd P efd8a37d6ac44bb4b4ef2f0e61cdb11c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65c308b30fa54aaba7bd5203d9b258a1" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40815 } } peers { permanent_uuid: "efd8a37d6ac44bb4b4ef2f0e61cdb11c" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45293 } } peers { permanent_uuid: "eecd298bb09548e0b4f4edffda7cbe91" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43223 } }
I20260621 07:47:42.691079 8206 raft_consensus.cc:374] T 60f981df8beb4d9492ea4fe1c4f40fbd P efd8a37d6ac44bb4b4ef2f0e61cdb11c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:42.691098 8206 raft_consensus.cc:729] T 60f981df8beb4d9492ea4fe1c4f40fbd P efd8a37d6ac44bb4b4ef2f0e61cdb11c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efd8a37d6ac44bb4b4ef2f0e61cdb11c, State: Initialized, Role: FOLLOWER
I20260621 07:47:42.691215 8206 consensus_queue.cc:260] T 60f981df8beb4d9492ea4fe1c4f40fbd P efd8a37d6ac44bb4b4ef2f0e61cdb11c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65c308b30fa54aaba7bd5203d9b258a1" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40815 } } peers { permanent_uuid: "efd8a37d6ac44bb4b4ef2f0e61cdb11c" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45293 } } peers { permanent_uuid: "eecd298bb09548e0b4f4edffda7cbe91" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43223 } }
I20260621 07:47:42.691547 8206 ts_tablet_manager.cc:1428] T 60f981df8beb4d9492ea4fe1c4f40fbd P efd8a37d6ac44bb4b4ef2f0e61cdb11c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260621 07:47:42.692065 8211 tablet_bootstrap.cc:492] T 60f981df8beb4d9492ea4fe1c4f40fbd P eecd298bb09548e0b4f4edffda7cbe91: Bootstrap starting.
I20260621 07:47:42.692368 8207 tablet_bootstrap.cc:492] T 60f981df8beb4d9492ea4fe1c4f40fbd P 65c308b30fa54aaba7bd5203d9b258a1: No bootstrap required, opened a new log
I20260621 07:47:42.692437 8207 ts_tablet_manager.cc:1397] T 60f981df8beb4d9492ea4fe1c4f40fbd P 65c308b30fa54aaba7bd5203d9b258a1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 07:47:42.692870 8211 tablet_bootstrap.cc:654] T 60f981df8beb4d9492ea4fe1c4f40fbd P eecd298bb09548e0b4f4edffda7cbe91: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:42.692803 8207 raft_consensus.cc:348] T 60f981df8beb4d9492ea4fe1c4f40fbd P 65c308b30fa54aaba7bd5203d9b258a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65c308b30fa54aaba7bd5203d9b258a1" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40815 } } peers { permanent_uuid: "efd8a37d6ac44bb4b4ef2f0e61cdb11c" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45293 } } peers { permanent_uuid: "eecd298bb09548e0b4f4edffda7cbe91" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43223 } }
I20260621 07:47:42.692904 8207 raft_consensus.cc:374] T 60f981df8beb4d9492ea4fe1c4f40fbd P 65c308b30fa54aaba7bd5203d9b258a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:42.692934 8207 raft_consensus.cc:729] T 60f981df8beb4d9492ea4fe1c4f40fbd P 65c308b30fa54aaba7bd5203d9b258a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c308b30fa54aaba7bd5203d9b258a1, State: Initialized, Role: FOLLOWER
I20260621 07:47:42.693068 8207 consensus_queue.cc:260] T 60f981df8beb4d9492ea4fe1c4f40fbd P 65c308b30fa54aaba7bd5203d9b258a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65c308b30fa54aaba7bd5203d9b258a1" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40815 } } peers { permanent_uuid: "efd8a37d6ac44bb4b4ef2f0e61cdb11c" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45293 } } peers { permanent_uuid: "eecd298bb09548e0b4f4edffda7cbe91" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43223 } }
I20260621 07:47:42.693418 8207 ts_tablet_manager.cc:1428] T 60f981df8beb4d9492ea4fe1c4f40fbd P 65c308b30fa54aaba7bd5203d9b258a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260621 07:47:42.693933 8212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:42.692506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:40701 (local address 127.0.192.193:37439)
0621 07:47:42.693777 (+ 1271us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:42.693827 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1196,"thread_start_us":104,"threads_started":1}
I20260621 07:47:42.694407 8215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:42.693782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:40701 (local address 127.0.192.194:38089)
0621 07:47:42.694264 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:42.694314 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":76,"threads_started":1}
I20260621 07:47:42.694682 7844 heartbeater.cc:503] Master 127.0.192.253:43105 was elected leader, sending a full tablet report...
I20260621 07:47:42.694875 7981 heartbeater.cc:503] Master 127.0.192.253:43105 was elected leader, sending a full tablet report...
I20260621 07:47:42.695322 8215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:42.695084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:40701 (local address 127.0.192.194:40147)
0621 07:47:42.695220 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:42.695266 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260621 07:47:42.696622 8211 tablet_bootstrap.cc:492] T 60f981df8beb4d9492ea4fe1c4f40fbd P eecd298bb09548e0b4f4edffda7cbe91: No bootstrap required, opened a new log
I20260621 07:47:42.696719 8211 ts_tablet_manager.cc:1397] T 60f981df8beb4d9492ea4fe1c4f40fbd P eecd298bb09548e0b4f4edffda7cbe91: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260621 07:47:42.697064 8211 raft_consensus.cc:348] T 60f981df8beb4d9492ea4fe1c4f40fbd P eecd298bb09548e0b4f4edffda7cbe91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65c308b30fa54aaba7bd5203d9b258a1" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40815 } } peers { permanent_uuid: "efd8a37d6ac44bb4b4ef2f0e61cdb11c" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45293 } } peers { permanent_uuid: "eecd298bb09548e0b4f4edffda7cbe91" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43223 } }
I20260621 07:47:42.697152 8211 raft_consensus.cc:374] T 60f981df8beb4d9492ea4fe1c4f40fbd P eecd298bb09548e0b4f4edffda7cbe91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:42.697175 8211 raft_consensus.cc:729] T 60f981df8beb4d9492ea4fe1c4f40fbd P eecd298bb09548e0b4f4edffda7cbe91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eecd298bb09548e0b4f4edffda7cbe91, State: Initialized, Role: FOLLOWER
I20260621 07:47:42.697304 8211 consensus_queue.cc:260] T 60f981df8beb4d9492ea4fe1c4f40fbd P eecd298bb09548e0b4f4edffda7cbe91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65c308b30fa54aaba7bd5203d9b258a1" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40815 } } peers { permanent_uuid: "efd8a37d6ac44bb4b4ef2f0e61cdb11c" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45293 } } peers { permanent_uuid: "eecd298bb09548e0b4f4edffda7cbe91" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43223 } }
I20260621 07:47:42.697559 8211 ts_tablet_manager.cc:1428] T 60f981df8beb4d9492ea4fe1c4f40fbd P eecd298bb09548e0b4f4edffda7cbe91: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260621 07:47:42.698300 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:42.697915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:40701 (local address 127.0.192.195:54457)
0621 07:47:42.698146 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:42.698204 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":96,"threads_started":1}
I20260621 07:47:42.698904 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:42.698720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:40701 (local address 127.0.192.195:51289)
0621 07:47:42.698803 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:42.698851 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260621 07:47:42.698912 8117 heartbeater.cc:503] Master 127.0.192.253:43105 was elected leader, sending a full tablet report...
I20260621 07:47:42.732978 8210 raft_consensus.cc:482] T 60f981df8beb4d9492ea4fe1c4f40fbd P efd8a37d6ac44bb4b4ef2f0e61cdb11c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:47:42.733115 8210 raft_consensus.cc:504] T 60f981df8beb4d9492ea4fe1c4f40fbd P efd8a37d6ac44bb4b4ef2f0e61cdb11c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65c308b30fa54aaba7bd5203d9b258a1" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40815 } } peers { permanent_uuid: "efd8a37d6ac44bb4b4ef2f0e61cdb11c" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45293 } } peers { permanent_uuid: "eecd298bb09548e0b4f4edffda7cbe91" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43223 } }
I20260621 07:47:42.734277 8210 leader_election.cc:288] T 60f981df8beb4d9492ea4fe1c4f40fbd P efd8a37d6ac44bb4b4ef2f0e61cdb11c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65c308b30fa54aaba7bd5203d9b258a1 (127.0.192.194:40815), eecd298bb09548e0b4f4edffda7cbe91 (127.0.192.195:43223)
I20260621 07:47:42.738888 8203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:42.734523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:42229 (local address 127.0.192.194:40815)
0621 07:47:42.734687 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:42.734693 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:42.734709 (+ 16us) server_negotiation.cc:407] Connection header received
0621 07:47:42.734761 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:42.734767 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:42.734843 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:42.734916 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:42.735261 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:42.736081 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:42.737562 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:42.738152 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:42.738285 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:42.738340 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:42.738687 (+ 347us) server_negotiation.cc:299] Negotiation successful
0621 07:47:42.738759 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 07:47:42.738888 8205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:42.734658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:36925 (local address 127.0.192.195:43223)
0621 07:47:42.734798 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:42.734802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:42.735008 (+ 206us) server_negotiation.cc:407] Connection header received
0621 07:47:42.735102 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:42.735105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:42.735162 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:42.735265 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:42.735482 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:42.736244 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:42.737562 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:42.738152 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:42.738285 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:42.738340 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:42.738678 (+ 338us) server_negotiation.cc:299] Negotiation successful
0621 07:47:42.738760 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260621 07:47:42.738925 8212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:42.734408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:40815 (local address 127.0.192.193:42229)
0621 07:47:42.734536 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:42.734552 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:42.734642 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:42.734980 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:42.734983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:42.735011 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:42.735148 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:42.735154 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:42.736210 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:42.736213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:42.737445 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:42.737455 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:42.738307 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:42.738311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:42.738489 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:42.738568 (+ 79us) client_negotiation.cc:769] Sending connection context
0621 07:47:42.738649 (+ 81us) client_negotiation.cc:240] Negotiation successful
0621 07:47:42.738757 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":164,"spinlock_wait_cycles":4096}
I20260621 07:47:42.739132 8218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:42.734546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.195:43223 (local address 127.0.192.193:36925)
0621 07:47:42.734859 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:42.734869 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:42.735013 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:42.735286 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:42.735288 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:42.735304 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:42.735381 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:42.735385 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:42.736328 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:42.736331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:42.737445 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:42.737455 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:42.738307 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:42.738311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:42.738528 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:42.738579 (+ 51us) client_negotiation.cc:769] Sending connection context
0621 07:47:42.738649 (+ 70us) client_negotiation.cc:240] Negotiation successful
0621 07:47:42.738757 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":111,"spinlock_wait_cycles":1664,"thread_start_us":125,"threads_started":1}
I20260621 07:47:42.739540 7933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60f981df8beb4d9492ea4fe1c4f40fbd" candidate_uuid: "efd8a37d6ac44bb4b4ef2f0e61cdb11c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c308b30fa54aaba7bd5203d9b258a1" is_pre_election: true
I20260621 07:47:42.739607 8069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60f981df8beb4d9492ea4fe1c4f40fbd" candidate_uuid: "efd8a37d6ac44bb4b4ef2f0e61cdb11c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eecd298bb09548e0b4f4edffda7cbe91" is_pre_election: true
I20260621 07:47:42.739727 8069 raft_consensus.cc:2454] T 60f981df8beb4d9492ea4fe1c4f40fbd P eecd298bb09548e0b4f4edffda7cbe91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efd8a37d6ac44bb4b4ef2f0e61cdb11c in term 0.
I20260621 07:47:42.739759 7933 raft_consensus.cc:2454] T 60f981df8beb4d9492ea4fe1c4f40fbd P 65c308b30fa54aaba7bd5203d9b258a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efd8a37d6ac44bb4b4ef2f0e61cdb11c in term 0.
I20260621 07:47:42.740134 7731 leader_election.cc:302] T 60f981df8beb4d9492ea4fe1c4f40fbd P efd8a37d6ac44bb4b4ef2f0e61cdb11c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eecd298bb09548e0b4f4edffda7cbe91, efd8a37d6ac44bb4b4ef2f0e61cdb11c; no voters:
I20260621 07:47:42.740362 8210 raft_consensus.cc:2781] T 60f981df8beb4d9492ea4fe1c4f40fbd P efd8a37d6ac44bb4b4ef2f0e61cdb11c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:47:42.740415 8210 raft_consensus.cc:482] T 60f981df8beb4d9492ea4fe1c4f40fbd P efd8a37d6ac44bb4b4ef2f0e61cdb11c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:47:42.740450 8210 raft_consensus.cc:3037] T 60f981df8beb4d9492ea4fe1c4f40fbd P efd8a37d6ac44bb4b4ef2f0e61cdb11c [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:42.741362 8210 raft_consensus.cc:504] T 60f981df8beb4d9492ea4fe1c4f40fbd P efd8a37d6ac44bb4b4ef2f0e61cdb11c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65c308b30fa54aaba7bd5203d9b258a1" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40815 } } peers { permanent_uuid: "efd8a37d6ac44bb4b4ef2f0e61cdb11c" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45293 } } peers { permanent_uuid: "eecd298bb09548e0b4f4edffda7cbe91" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43223 } }
I20260621 07:47:42.741617 8210 leader_election.cc:288] T 60f981df8beb4d9492ea4fe1c4f40fbd P efd8a37d6ac44bb4b4ef2f0e61cdb11c [CANDIDATE]: Term 1 election: Requested vote from peers 65c308b30fa54aaba7bd5203d9b258a1 (127.0.192.194:40815), eecd298bb09548e0b4f4edffda7cbe91 (127.0.192.195:43223)
I20260621 07:47:42.741894 7933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60f981df8beb4d9492ea4fe1c4f40fbd" candidate_uuid: "efd8a37d6ac44bb4b4ef2f0e61cdb11c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c308b30fa54aaba7bd5203d9b258a1"
I20260621 07:47:42.741935 8069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60f981df8beb4d9492ea4fe1c4f40fbd" candidate_uuid: "efd8a37d6ac44bb4b4ef2f0e61cdb11c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eecd298bb09548e0b4f4edffda7cbe91"
I20260621 07:47:42.742031 7933 raft_consensus.cc:3037] T 60f981df8beb4d9492ea4fe1c4f40fbd P 65c308b30fa54aaba7bd5203d9b258a1 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:42.742046 8069 raft_consensus.cc:3037] T 60f981df8beb4d9492ea4fe1c4f40fbd P eecd298bb09548e0b4f4edffda7cbe91 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:42.742813 7933 raft_consensus.cc:2454] T 60f981df8beb4d9492ea4fe1c4f40fbd P 65c308b30fa54aaba7bd5203d9b258a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efd8a37d6ac44bb4b4ef2f0e61cdb11c in term 1.
I20260621 07:47:42.742839 8069 raft_consensus.cc:2454] T 60f981df8beb4d9492ea4fe1c4f40fbd P eecd298bb09548e0b4f4edffda7cbe91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efd8a37d6ac44bb4b4ef2f0e61cdb11c in term 1.
I20260621 07:47:42.743083 7730 leader_election.cc:302] T 60f981df8beb4d9492ea4fe1c4f40fbd P efd8a37d6ac44bb4b4ef2f0e61cdb11c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c308b30fa54aaba7bd5203d9b258a1, efd8a37d6ac44bb4b4ef2f0e61cdb11c; no voters:
I20260621 07:47:42.743255 8210 raft_consensus.cc:2781] T 60f981df8beb4d9492ea4fe1c4f40fbd P efd8a37d6ac44bb4b4ef2f0e61cdb11c [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:47:42.743430 8210 raft_consensus.cc:686] T 60f981df8beb4d9492ea4fe1c4f40fbd P efd8a37d6ac44bb4b4ef2f0e61cdb11c [term 1 LEADER]: Becoming Leader. State: Replica: efd8a37d6ac44bb4b4ef2f0e61cdb11c, State: Running, Role: LEADER
I20260621 07:47:42.743647 8210 consensus_queue.cc:237] T 60f981df8beb4d9492ea4fe1c4f40fbd P efd8a37d6ac44bb4b4ef2f0e61cdb11c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65c308b30fa54aaba7bd5203d9b258a1" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40815 } } peers { permanent_uuid: "efd8a37d6ac44bb4b4ef2f0e61cdb11c" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45293 } } peers { permanent_uuid: "eecd298bb09548e0b4f4edffda7cbe91" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43223 } }
I20260621 07:47:42.744127 8218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:42.743916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:40701 (local address 127.0.192.193:59253)
0621 07:47:42.744016 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:42.744070 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260621 07:47:42.745625 7471 catalog_manager.cc:5495] T 60f981df8beb4d9492ea4fe1c4f40fbd P efd8a37d6ac44bb4b4ef2f0e61cdb11c reported cstate change: term changed from 0 to 1, leader changed from <none> to efd8a37d6ac44bb4b4ef2f0e61cdb11c (127.0.192.193). New cstate: current_term: 1 leader_uuid: "efd8a37d6ac44bb4b4ef2f0e61cdb11c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65c308b30fa54aaba7bd5203d9b258a1" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efd8a37d6ac44bb4b4ef2f0e61cdb11c" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 45293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eecd298bb09548e0b4f4edffda7cbe91" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43223 } health_report { overall_health: UNKNOWN } } }
I20260621 07:47:42.762290 8204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:42.759003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51764 (local address 127.0.192.193:45293)
0621 07:47:42.759136 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:42.759143 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:42.759206 (+ 63us) server_negotiation.cc:407] Connection header received
0621 07:47:42.759361 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:42.759364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:42.759410 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:42.759514 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:47:42.759759 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:42.760247 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:42.761075 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:42.761285 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:42.761380 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:42.761608 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:47:42.761971 (+ 363us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:47:42.762080 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:42.762132 (+ 52us) server_negotiation.cc:299] Negotiation successful
0621 07:47:42.762174 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 07:47:42.764671 8069 raft_consensus.cc:1261] T 60f981df8beb4d9492ea4fe1c4f40fbd P eecd298bb09548e0b4f4edffda7cbe91 [term 1 FOLLOWER]: Refusing update from remote peer efd8a37d6ac44bb4b4ef2f0e61cdb11c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:47:42.764899 7933 raft_consensus.cc:1261] T 60f981df8beb4d9492ea4fe1c4f40fbd P 65c308b30fa54aaba7bd5203d9b258a1 [term 1 FOLLOWER]: Refusing update from remote peer efd8a37d6ac44bb4b4ef2f0e61cdb11c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:47:42.765121 8210 consensus_queue.cc:1035] T 60f981df8beb4d9492ea4fe1c4f40fbd P efd8a37d6ac44bb4b4ef2f0e61cdb11c [LEADER]: Connected to new peer: Peer: permanent_uuid: "eecd298bb09548e0b4f4edffda7cbe91" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:47:42.765422 8219 consensus_queue.cc:1035] T 60f981df8beb4d9492ea4fe1c4f40fbd P efd8a37d6ac44bb4b4ef2f0e61cdb11c [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c308b30fa54aaba7bd5203d9b258a1" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 40815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:47:42.766806 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:42.766533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:40701 (local address 127.0.192.195:39011)
0621 07:47:42.766671 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:42.766744 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260621 07:47:42.768963 8215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:42.768745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:40701 (local address 127.0.192.194:45527)
0621 07:47:42.768833 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:42.768887 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260621 07:47:42.770478 8215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:42.770276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:40701 (local address 127.0.192.194:44509)
0621 07:47:42.770387 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:42.770428 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260621 07:47:42.771250 8218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:42.770638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:40701 (local address 127.0.192.193:58205)
0621 07:47:42.771119 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:42.771167 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421}
I20260621 07:47:42.771677 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:42.771452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:40701 (local address 127.0.192.195:57103)
0621 07:47:42.771560 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:42.771601 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:40701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260621 07:47:42.776515 7470 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:47:42.776662 7470 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:47:42.779721 7470 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b7be1241e6d426d87e026a0d4b8292c: Sending DeleteTablet for 3 replicas of tablet 60f981df8beb4d9492ea4fe1c4f40fbd
I20260621 07:47:42.780226 7913 tablet_service.cc:1514] Processing DeleteTablet for tablet 60f981df8beb4d9492ea4fe1c4f40fbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:47:42 UTC) from {username='oryx', principal='oryx/127.0.192.253@KRBTEST.COM'} at 127.0.0.1:41036
I20260621 07:47:42.780442 7776 tablet_service.cc:1514] Processing DeleteTablet for tablet 60f981df8beb4d9492ea4fe1c4f40fbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:47:42 UTC) from {username='oryx', principal='oryx/127.0.192.253@KRBTEST.COM'} at 127.0.0.1:51754
I20260621 07:47:42.780951 8049 tablet_service.cc:1514] Processing DeleteTablet for tablet 60f981df8beb4d9492ea4fe1c4f40fbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:47:42 UTC) from {username='oryx', principal='oryx/127.0.192.253@KRBTEST.COM'} at 127.0.0.1:43314
I20260621 07:47:42.781200 8233 tablet_replica.cc:330] T 60f981df8beb4d9492ea4fe1c4f40fbd P efd8a37d6ac44bb4b4ef2f0e61cdb11c: stopping tablet replica
I20260621 07:47:42.781219 8234 tablet_replica.cc:330] T 60f981df8beb4d9492ea4fe1c4f40fbd P eecd298bb09548e0b4f4edffda7cbe91: stopping tablet replica
I20260621 07:47:42.781342 8233 raft_consensus.cc:2229] T 60f981df8beb4d9492ea4fe1c4f40fbd P efd8a37d6ac44bb4b4ef2f0e61cdb11c [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:47:42.781430 8234 raft_consensus.cc:2229] T 60f981df8beb4d9492ea4fe1c4f40fbd P eecd298bb09548e0b4f4edffda7cbe91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:47:42.781559 8234 raft_consensus.cc:2258] T 60f981df8beb4d9492ea4fe1c4f40fbd P eecd298bb09548e0b4f4edffda7cbe91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:47:42.781574 8233 raft_consensus.cc:2258] T 60f981df8beb4d9492ea4fe1c4f40fbd P efd8a37d6ac44bb4b4ef2f0e61cdb11c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:47:42.781783 8232 tablet_replica.cc:330] T 60f981df8beb4d9492ea4fe1c4f40fbd P 65c308b30fa54aaba7bd5203d9b258a1: stopping tablet replica
I20260621 07:47:42.781854 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 7616
I20260621 07:47:42.781893 8232 raft_consensus.cc:2229] T 60f981df8beb4d9492ea4fe1c4f40fbd P 65c308b30fa54aaba7bd5203d9b258a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:47:42.784161 8232 raft_consensus.cc:2258] T 60f981df8beb4d9492ea4fe1c4f40fbd P 65c308b30fa54aaba7bd5203d9b258a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:47:42.784834 8232 ts_tablet_manager.cc:1905] T 60f981df8beb4d9492ea4fe1c4f40fbd P 65c308b30fa54aaba7bd5203d9b258a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:47:42.786708 8232 ts_tablet_manager.cc:1918] T 60f981df8beb4d9492ea4fe1c4f40fbd P 65c308b30fa54aaba7bd5203d9b258a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:47:42.786791 8232 log.cc:1196] T 60f981df8beb4d9492ea4fe1c4f40fbd P 65c308b30fa54aaba7bd5203d9b258a1: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-1/wal/wals/60f981df8beb4d9492ea4fe1c4f40fbd
I20260621 07:47:42.787030 8232 ts_tablet_manager.cc:1939] T 60f981df8beb4d9492ea4fe1c4f40fbd P 65c308b30fa54aaba7bd5203d9b258a1: Deleting consensus metadata
I20260621 07:47:42.787767 8234 ts_tablet_manager.cc:1905] T 60f981df8beb4d9492ea4fe1c4f40fbd P eecd298bb09548e0b4f4edffda7cbe91: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260621 07:47:42.787853 8202 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:42.787560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:45293 (local address 127.0.0.1:51772)
0621 07:47:42.787705 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:42.787749 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.193:45293: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":70}
I20260621 07:47:42.787904 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 7850
I20260621 07:47:42.787948 7456 catalog_manager.cc:4841] TS 65c308b30fa54aaba7bd5203d9b258a1 (127.0.192.194:40815): tablet 60f981df8beb4d9492ea4fe1c4f40fbd (table test-table [id=6a51804283a44561a61978f376783080]) successfully deleted
W20260621 07:47:42.789000 7458 catalog_manager.cc:4584] TS efd8a37d6ac44bb4b4ef2f0e61cdb11c (127.0.192.193:45293): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 60f981df8beb4d9492ea4fe1c4f40fbd: Network error: Client connection negotiation failed: client connection to 127.0.192.193:45293: connect: Connection reset by peer (error 104)
I20260621 07:47:42.792102 8234 ts_tablet_manager.cc:1918] T 60f981df8beb4d9492ea4fe1c4f40fbd P eecd298bb09548e0b4f4edffda7cbe91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:47:42.792178 8234 log.cc:1196] T 60f981df8beb4d9492ea4fe1c4f40fbd P eecd298bb09548e0b4f4edffda7cbe91: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782028028522580-771-0/minicluster-data/ts-2/wal/wals/60f981df8beb4d9492ea4fe1c4f40fbd
I20260621 07:47:42.792400 8234 ts_tablet_manager.cc:1939] T 60f981df8beb4d9492ea4fe1c4f40fbd P eecd298bb09548e0b4f4edffda7cbe91: Deleting consensus metadata
I20260621 07:47:42.793085 7457 catalog_manager.cc:4841] TS eecd298bb09548e0b4f4edffda7cbe91 (127.0.192.195:43223): tablet 60f981df8beb4d9492ea4fe1c4f40fbd (table test-table [id=6a51804283a44561a61978f376783080]) successfully deleted
I20260621 07:47:42.794965 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 7986
I20260621 07:47:42.800040 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 7335
I20260621 07:47:42.805927 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 7443
2026-06-21T07:47:42Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4845 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:47:42 dist-test-slave-lzx3 krb5kdc[8237](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:47:42 dist-test-slave-lzx3 krb5kdc[8237](info): set up 2 sockets
Jun 21 07:47:42 dist-test-slave-lzx3 krb5kdc[8237](info): commencing operation
krb5kdc: starting...
W20260621 07:47:43.764923 771 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/krb5kdc/test-user.keytab.
Jun 21 07:47:43 dist-test-slave-lzx3 krb5kdc[8237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:47:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:47:43Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.192.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.192.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.192.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/krb5kdc/oryx_127.0.192.254.keytab.
Entry for principal oryx/127.0.192.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/krb5kdc/oryx_127.0.192.254.keytab.
I20260621 07:47:43.870621 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:45399
--webserver_interface=127.0.192.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/krb5kdc/oryx_127.0.192.254.keytab
--principal=oryx/127.0.192.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:38987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:45399
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:43.972128 8250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:43.972348 8250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:43.972385 8250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:43.976293 8250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:47:43.976353 8250 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260621 07:47:43.976380 8250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:43.976397 8250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:43.976410 8250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:47:43.976424 8250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:47:43.981170 8250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:38987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:45399
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/krb5kdc/oryx_127.0.192.254.keytab
--principal=oryx/127.0.192.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:45399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:43.982026 8250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:43.982872 8250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:43.990597 8256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:43.990613 8255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:43.990780 8250 server_base.cc:1034] running on GCE node
W20260621 07:47:43.990615 8258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:43.991379 8250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:43.992393 8250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:43.993544 8250 hybrid_clock.cc:648] HybridClock initialized: now 1782028063993519 us; error 35 us; skew 500 ppm
Jun 21 07:47:43 dist-test-slave-lzx3 krb5kdc[8237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028063, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.192.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:43.996032 8250 init.cc:377] Logged in from keytab as oryx/127.0.192.254@KRBTEST.COM (short username oryx)
I20260621 07:47:43.996837 8250 webserver.cc:460] Webserver started at http://127.0.192.254:42647/ using document root <none> and password file <none>
I20260621 07:47:43.997349 8250 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:43.997434 8250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:43.997644 8250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:43.999496 8250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "7a5a7e6a80b74aebbeb361add74240ab"
format_stamp: "Formatted at 2026-06-21 07:47:43 on dist-test-slave-lzx3"
server_key: "e948937ce620da4377ac0ed0eede44b1"
server_key_iv: "f87c91cae192e81505de38e2ed76b129"
server_key_version: "encryptionkey@0"
I20260621 07:47:44.000051 8250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "7a5a7e6a80b74aebbeb361add74240ab"
format_stamp: "Formatted at 2026-06-21 07:47:43 on dist-test-slave-lzx3"
server_key: "e948937ce620da4377ac0ed0eede44b1"
server_key_iv: "f87c91cae192e81505de38e2ed76b129"
server_key_version: "encryptionkey@0"
I20260621 07:47:44.003410 8250 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260621 07:47:44.005533 8265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:44.006716 8250 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 07:47:44.006846 8250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "7a5a7e6a80b74aebbeb361add74240ab"
format_stamp: "Formatted at 2026-06-21 07:47:43 on dist-test-slave-lzx3"
server_key: "e948937ce620da4377ac0ed0eede44b1"
server_key_iv: "f87c91cae192e81505de38e2ed76b129"
server_key_version: "encryptionkey@0"
I20260621 07:47:44.006964 8250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:44.045070 8250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:44.047880 8250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:44.048077 8250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:44.055910 8250 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:45399
I20260621 07:47:44.055933 8327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:45399 every 8 connection(s)
I20260621 07:47:44.056850 8250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:47:44.059502 8328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:44.064492 8328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a5a7e6a80b74aebbeb361add74240ab: Bootstrap starting.
I20260621 07:47:44.065552 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 8250
I20260621 07:47:44.065680 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:47:44.066006 771 external_mini_cluster.cc:1428] Setting key c362b956cc0af0695d8624fac4f46e9b
I20260621 07:47:44.067305 8328 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a5a7e6a80b74aebbeb361add74240ab: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:44.068010 8328 log.cc:824] T 00000000000000000000000000000000 P 7a5a7e6a80b74aebbeb361add74240ab: Log is configured to *not* fsync() on all Append() calls
I20260621 07:47:44.069839 8328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a5a7e6a80b74aebbeb361add74240ab: No bootstrap required, opened a new log
I20260621 07:47:44.073097 8328 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a5a7e6a80b74aebbeb361add74240ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5a7e6a80b74aebbeb361add74240ab" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45399 } }
I20260621 07:47:44.073361 8328 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a5a7e6a80b74aebbeb361add74240ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:44.073433 8328 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a5a7e6a80b74aebbeb361add74240ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a5a7e6a80b74aebbeb361add74240ab, State: Initialized, Role: FOLLOWER
I20260621 07:47:44.073858 8328 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a5a7e6a80b74aebbeb361add74240ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5a7e6a80b74aebbeb361add74240ab" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45399 } }
I20260621 07:47:44.073994 8328 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a5a7e6a80b74aebbeb361add74240ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:47:44.074072 8328 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a5a7e6a80b74aebbeb361add74240ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:47:44.074162 8328 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a5a7e6a80b74aebbeb361add74240ab [term 0 FOLLOWER]: Advancing to term 1
Jun 21 07:47:44 dist-test-slave-lzx3 krb5kdc[8237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM
I20260621 07:47:44.075230 8328 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a5a7e6a80b74aebbeb361add74240ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5a7e6a80b74aebbeb361add74240ab" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45399 } }
I20260621 07:47:44.075573 8328 leader_election.cc:302] T 00000000000000000000000000000000 P 7a5a7e6a80b74aebbeb361add74240ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a5a7e6a80b74aebbeb361add74240ab; no voters:
I20260621 07:47:44.075928 8328 leader_election.cc:288] T 00000000000000000000000000000000 P 7a5a7e6a80b74aebbeb361add74240ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:47:44.076037 8333 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a5a7e6a80b74aebbeb361add74240ab [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:47:44.076251 8333 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a5a7e6a80b74aebbeb361add74240ab [term 1 LEADER]: Becoming Leader. State: Replica: 7a5a7e6a80b74aebbeb361add74240ab, State: Running, Role: LEADER
I20260621 07:47:44.076550 8333 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a5a7e6a80b74aebbeb361add74240ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5a7e6a80b74aebbeb361add74240ab" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45399 } }
I20260621 07:47:44.077190 8328 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a5a7e6a80b74aebbeb361add74240ab [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:47:44.078140 8334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a5a7e6a80b74aebbeb361add74240ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a5a7e6a80b74aebbeb361add74240ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5a7e6a80b74aebbeb361add74240ab" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45399 } } }
I20260621 07:47:44.078258 8334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a5a7e6a80b74aebbeb361add74240ab [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:44.078554 8335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a5a7e6a80b74aebbeb361add74240ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a5a7e6a80b74aebbeb361add74240ab. Latest consensus state: current_term: 1 leader_uuid: "7a5a7e6a80b74aebbeb361add74240ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5a7e6a80b74aebbeb361add74240ab" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45399 } } }
I20260621 07:47:44.078645 8335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a5a7e6a80b74aebbeb361add74240ab [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:44.079572 8342 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:47:44.080610 8331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:44.067112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46452 (local address 127.0.192.254:45399)
0621 07:47:44.067558 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:44.067572 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:44.067605 (+ 33us) server_negotiation.cc:407] Connection header received
0621 07:47:44.068264 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:44.068281 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:44.068726 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:44.069085 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:44.069862 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.070788 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.071597 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.072061 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.072227 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:44.074810 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:44.074815 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:44.074826 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:44.074853 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:44.076793 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:44.077267 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:44.077272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:44.077278 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:44.077415 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:44.077744 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:44.077748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:44.077749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:44.078292 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:44.078483 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:44.079395 (+ 912us) server_negotiation.cc:299] Negotiation successful
0621 07:47:44.079678 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":139,"threads_started":1}
I20260621 07:47:44.082324 8342 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:47:44.087673 8342 catalog_manager.cc:1347] Generated new cluster ID: b6fa94ac56c6466584db9283a7418928
I20260621 07:47:44.087742 8342 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:47:44.096712 8342 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:47:44.097774 8342 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:47:44.108004 8342 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a5a7e6a80b74aebbeb361add74240ab: Generated new TSK 0
I20260621 07:47:44.108762 8342 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.192.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.192.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.192.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/krb5kdc/oryx_127.0.192.193.keytab.
Entry for principal oryx/127.0.192.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/krb5kdc/oryx_127.0.192.193.keytab.
I20260621 07:47:44.143931 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.193:0
--local_ip_for_outbound_sockets=127.0.192.193
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:45399
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/krb5kdc/oryx_127.0.192.193.keytab
--principal=oryx/127.0.192.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:38987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:44.246999 8354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:44.247203 8354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:44.247241 8354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:44.251046 8354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:44.251096 8354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:44.251165 8354 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 07:47:44.251245 8354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.193
I20260621 07:47:44.256235 8354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:38987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/krb5kdc/oryx_127.0.192.193.keytab
--principal=oryx/127.0.192.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.192.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.192.254:45399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.193
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:44.257153 8354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:44.258095 8354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:44.268380 8359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:44.268460 8360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:44.268699 8362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:44.269001 8354 server_base.cc:1034] running on GCE node
I20260621 07:47:44.269393 8354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:44.269948 8354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:44.271121 8354 hybrid_clock.cc:648] HybridClock initialized: now 1782028064271089 us; error 43 us; skew 500 ppm
Jun 21 07:47:44 dist-test-slave-lzx3 krb5kdc[8237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028064, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.192.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:44.273756 8354 init.cc:377] Logged in from keytab as oryx/127.0.192.193@KRBTEST.COM (short username oryx)
I20260621 07:47:44.274540 8354 webserver.cc:460] Webserver started at http://127.0.192.193:36455/ using document root <none> and password file <none>
I20260621 07:47:44.275045 8354 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:44.275149 8354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:44.275395 8354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:44.277416 8354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-0/data/instance:
uuid: "73148a30e9e54a29b9ab2894b5f3f8ae"
format_stamp: "Formatted at 2026-06-21 07:47:44 on dist-test-slave-lzx3"
server_key: "f206952c1f95d697aab66a38e7c80e46"
server_key_iv: "9ad752d804bd150f73223d4e5bcdfba9"
server_key_version: "encryptionkey@0"
I20260621 07:47:44.277977 8354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-0/wal/instance:
uuid: "73148a30e9e54a29b9ab2894b5f3f8ae"
format_stamp: "Formatted at 2026-06-21 07:47:44 on dist-test-slave-lzx3"
server_key: "f206952c1f95d697aab66a38e7c80e46"
server_key_iv: "9ad752d804bd150f73223d4e5bcdfba9"
server_key_version: "encryptionkey@0"
I20260621 07:47:44.281415 8354 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:47:44.283565 8369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:44.284634 8354 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260621 07:47:44.284750 8354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-0/wal
uuid: "73148a30e9e54a29b9ab2894b5f3f8ae"
format_stamp: "Formatted at 2026-06-21 07:47:44 on dist-test-slave-lzx3"
server_key: "f206952c1f95d697aab66a38e7c80e46"
server_key_iv: "9ad752d804bd150f73223d4e5bcdfba9"
server_key_version: "encryptionkey@0"
I20260621 07:47:44.284848 8354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:44.293228 8354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:44.296043 8354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:44.296223 8354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:44.297580 8354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:47:44.297647 8354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:44.297708 8354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:47:44.297760 8354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 07:47:44 dist-test-slave-lzx3 krb5kdc[8237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028064, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.192.193@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM
I20260621 07:47:44.308349 8354 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.193:33397
I20260621 07:47:44.308434 8484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.193:33397 every 8 connection(s)
I20260621 07:47:44.309203 8354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
I20260621 07:47:44.310667 8331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:44.298189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:51307 (local address 127.0.192.254:45399)
0621 07:47:44.298324 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:44.298329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:44.299198 (+ 869us) server_negotiation.cc:407] Connection header received
0621 07:47:44.300350 (+ 1152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:44.300353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:44.300405 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:44.300492 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:44.301692 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.302295 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.303595 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.303818 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.303928 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:44.306964 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:44.306970 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:44.306973 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:44.307003 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:44.308491 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:44.308992 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:44.308997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:44.308999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:44.309080 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:44.309482 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:44.309487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:44.309490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:44.309702 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:44.309824 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:44.310363 (+ 539us) server_negotiation.cc:299] Negotiation successful
0621 07:47:44.310501 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 07:47:44.311146 8378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:44.298583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45399 (local address 127.0.192.193:51307)
0621 07:47:44.299028 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:44.299081 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:44.300109 (+ 1028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:44.300777 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:44.300783 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:44.301130 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:44.301538 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:44.301550 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.302416 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.302421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:44.303463 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:44.303477 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.303967 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.303971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:44.304129 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:44.304720 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:44.304734 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:44.306791 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:44.308633 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:44.308638 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:44.308651 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:44.308888 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:44.309174 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:44.309179 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:44.309181 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:44.309361 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:44.309871 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:44.309877 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:44.310137 (+ 260us) client_negotiation.cc:769] Sending connection context
0621 07:47:44.310308 (+ 171us) client_negotiation.cc:240] Negotiation successful
0621 07:47:44.310523 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":100,"threads_started":1}
I20260621 07:47:44.312301 8485 heartbeater.cc:347] Connected to a master server at 127.0.192.254:45399
I20260621 07:47:44.312479 8485 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:44.312939 8485 heartbeater.cc:511] Master 127.0.192.254:45399 requested a full tablet report, sending...
I20260621 07:47:44.314265 8281 ts_manager.cc:194] Registered new tserver with Master: 73148a30e9e54a29b9ab2894b5f3f8ae (127.0.192.193:33397)
I20260621 07:47:44.315371 8281 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.192.193@KRBTEST.COM'} at 127.0.192.193:51307
I20260621 07:47:44.318935 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 8354
I20260621 07:47:44.319085 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-0/wal/instance
I20260621 07:47:44.319451 771 external_mini_cluster.cc:1428] Setting key d82cbf0635bffcbd809c4012cde2246c
I20260621 07:47:44.321285 8331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:44.314293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:35235 (local address 127.0.192.254:45399)
0621 07:47:44.314408 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:44.314414 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:44.314596 (+ 182us) server_negotiation.cc:407] Connection header received
0621 07:47:44.314703 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:44.314706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:44.314743 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:44.314832 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:44.315302 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.316006 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.317140 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.317334 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.317435 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:44.318431 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:44.318434 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:44.318436 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:44.318452 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:44.319820 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:44.320207 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:44.320210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:44.320212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:44.320284 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:44.320515 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:44.320518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:44.320519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:44.320705 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:44.320804 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:44.321048 (+ 244us) server_negotiation.cc:299] Negotiation successful
0621 07:47:44.321145 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 07:47:44.321296 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:44.314222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45399 (local address 127.0.192.193:35235)
0621 07:47:44.314522 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:44.314532 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:44.314632 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:44.314843 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:44.314846 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:44.315027 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:44.315176 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:44.315184 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.316134 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.316137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:44.316995 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:44.317003 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.317464 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.317467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:44.317556 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:44.318008 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:44.318019 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:44.318318 (+ 299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:44.319927 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:44.319930 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:44.319933 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:44.320122 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:44.320368 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:44.320370 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:44.320372 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:44.320437 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:44.320808 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:44.320811 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:44.320898 (+ 87us) client_negotiation.cc:769] Sending connection context
0621 07:47:44.320999 (+ 101us) client_negotiation.cc:240] Negotiation successful
0621 07:47:44.321126 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for oryx/127.0.192.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.192.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.192.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/krb5kdc/oryx_127.0.192.194.keytab.
Entry for principal oryx/127.0.192.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/krb5kdc/oryx_127.0.192.194.keytab.
I20260621 07:47:44.343181 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.194:0
--local_ip_for_outbound_sockets=127.0.192.194
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:45399
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/krb5kdc/oryx_127.0.192.194.keytab
--principal=oryx/127.0.192.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:38987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:44.447815 8494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:44.448028 8494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:44.448066 8494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:44.451962 8494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:44.452008 8494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:44.452041 8494 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 07:47:44.452068 8494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.194
I20260621 07:47:44.456785 8494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:38987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/krb5kdc/oryx_127.0.192.194.keytab
--principal=oryx/127.0.192.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.192.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.192.254:45399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.194
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:44.457736 8494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:44.458648 8494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:44.467417 8499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:44.467465 8494 server_base.cc:1034] running on GCE node
W20260621 07:47:44.467404 8502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:44.467404 8500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:44.467986 8494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:44.468598 8494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:44.469771 8494 hybrid_clock.cc:648] HybridClock initialized: now 1782028064469751 us; error 31 us; skew 500 ppm
Jun 21 07:47:44 dist-test-slave-lzx3 krb5kdc[8237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028064, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.192.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:44.472301 8494 init.cc:377] Logged in from keytab as oryx/127.0.192.194@KRBTEST.COM (short username oryx)
I20260621 07:47:44.473124 8494 webserver.cc:460] Webserver started at http://127.0.192.194:45283/ using document root <none> and password file <none>
I20260621 07:47:44.473580 8494 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:44.473632 8494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:44.473774 8494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:44.475682 8494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-1/data/instance:
uuid: "0f20308d9e324531a57e487bd730ba8f"
format_stamp: "Formatted at 2026-06-21 07:47:44 on dist-test-slave-lzx3"
server_key: "625dad73c264c5ac708a69d3a44401d4"
server_key_iv: "364e112e9684dde7635f9dd95ab774ad"
server_key_version: "encryptionkey@0"
I20260621 07:47:44.476163 8494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-1/wal/instance:
uuid: "0f20308d9e324531a57e487bd730ba8f"
format_stamp: "Formatted at 2026-06-21 07:47:44 on dist-test-slave-lzx3"
server_key: "625dad73c264c5ac708a69d3a44401d4"
server_key_iv: "364e112e9684dde7635f9dd95ab774ad"
server_key_version: "encryptionkey@0"
I20260621 07:47:44.479817 8494 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260621 07:47:44.481937 8509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:44.483038 8494 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 07:47:44.483162 8494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-1/wal
uuid: "0f20308d9e324531a57e487bd730ba8f"
format_stamp: "Formatted at 2026-06-21 07:47:44 on dist-test-slave-lzx3"
server_key: "625dad73c264c5ac708a69d3a44401d4"
server_key_iv: "364e112e9684dde7635f9dd95ab774ad"
server_key_version: "encryptionkey@0"
I20260621 07:47:44.483256 8494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:44.495380 8494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:44.498035 8494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:44.498232 8494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:44.499481 8494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:47:44.499549 8494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:44.499614 8494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:47:44.499657 8494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 07:47:44 dist-test-slave-lzx3 krb5kdc[8237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028064, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.192.194@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM
I20260621 07:47:44.510154 8494 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.194:43285
I20260621 07:47:44.510248 8624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.194:43285 every 8 connection(s)
I20260621 07:47:44.511093 8494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
I20260621 07:47:44.512810 8331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:44.500250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:38401 (local address 127.0.192.254:45399)
0621 07:47:44.500384 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:44.500388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:44.501207 (+ 819us) server_negotiation.cc:407] Connection header received
0621 07:47:44.502103 (+ 896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:44.502106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:44.502160 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:44.502219 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:44.503840 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.504723 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.506090 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.506318 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.506430 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:44.508867 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:44.508871 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:44.508874 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:44.508900 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:44.510465 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:44.511040 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:44.511045 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:44.511047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:44.511129 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:44.511532 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:44.511536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:44.511538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:44.511837 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:44.511979 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:44.512456 (+ 477us) server_negotiation.cc:299] Negotiation successful
0621 07:47:44.512641 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 07:47:44.513489 8518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:44.500650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45399 (local address 127.0.192.194:38401)
0621 07:47:44.501057 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:44.501101 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:44.501866 (+ 765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:44.502636 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:44.502644 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:44.503144 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:44.503680 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:44.503698 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.504855 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.504859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:44.505979 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:44.505987 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.506473 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.506476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:44.506638 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:44.507160 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:44.507177 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:44.508686 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:44.510604 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:44.510609 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:44.510620 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:44.510935 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:44.511251 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:44.511254 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:44.511256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:44.511421 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:44.511973 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:44.511978 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:44.512237 (+ 259us) client_negotiation.cc:769] Sending connection context
0621 07:47:44.512414 (+ 177us) client_negotiation.cc:240] Negotiation successful
0621 07:47:44.512745 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":118,"threads_started":1}
I20260621 07:47:44.514576 8625 heartbeater.cc:347] Connected to a master server at 127.0.192.254:45399
I20260621 07:47:44.514698 8625 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:44.515187 8625 heartbeater.cc:511] Master 127.0.192.254:45399 requested a full tablet report, sending...
I20260621 07:47:44.516187 8282 ts_manager.cc:194] Registered new tserver with Master: 0f20308d9e324531a57e487bd730ba8f (127.0.192.194:43285)
I20260621 07:47:44.516995 8282 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.192.194@KRBTEST.COM'} at 127.0.192.194:38401
I20260621 07:47:44.518498 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 8494
I20260621 07:47:44.518631 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-1/wal/instance
I20260621 07:47:44.518900 771 external_mini_cluster.cc:1428] Setting key 48778759e84eef865aa043f98e6e2bfe
I20260621 07:47:44.523773 8331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:44.516709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:35275 (local address 127.0.192.254:45399)
0621 07:47:44.516822 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:44.516826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:44.517018 (+ 192us) server_negotiation.cc:407] Connection header received
0621 07:47:44.517130 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:44.517133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:44.517182 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:44.517270 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:44.517676 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.518143 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.519126 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.519408 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.519508 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:44.520563 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:44.520567 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:44.520569 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:44.520621 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:44.522097 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:44.522713 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:44.522716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:44.522717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:44.522785 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:44.523070 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:44.523072 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:44.523073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:44.523238 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:44.523310 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:44.523602 (+ 292us) server_negotiation.cc:299] Negotiation successful
0621 07:47:44.523679 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260621 07:47:44.523962 8631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:44.516628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45399 (local address 127.0.192.194:35275)
0621 07:47:44.516934 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:44.516948 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:44.517041 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:44.517291 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:44.517293 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:44.517481 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:44.517561 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:44.517574 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.518303 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.518307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:44.519001 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:44.519011 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.519513 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.519516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:44.519618 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:44.520072 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:44.520085 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:44.520418 (+ 333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:44.522343 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:44.522348 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:44.522352 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:44.522622 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:44.522875 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:44.522880 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:44.522883 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:44.522982 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:44.523331 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:44.523336 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:44.523448 (+ 112us) client_negotiation.cc:769] Sending connection context
0621 07:47:44.523558 (+ 110us) client_negotiation.cc:240] Negotiation successful
0621 07:47:44.523706 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":85,"threads_started":1}
WARNING: no policy specified for oryx/127.0.192.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.192.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.192.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/krb5kdc/oryx_127.0.192.195.keytab.
Entry for principal oryx/127.0.192.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/krb5kdc/oryx_127.0.192.195.keytab.
I20260621 07:47:44.543534 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.195:0
--local_ip_for_outbound_sockets=127.0.192.195
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:45399
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/krb5kdc/oryx_127.0.192.195.keytab
--principal=oryx/127.0.192.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:38987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:44.647755 8634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:44.647989 8634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:44.648030 8634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:44.651877 8634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:44.651930 8634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:44.651963 8634 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 07:47:44.651990 8634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.195
I20260621 07:47:44.656814 8634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:38987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/krb5kdc/oryx_127.0.192.195.keytab
--principal=oryx/127.0.192.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.192.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.192.254:45399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.195
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:44.657590 8634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:44.658509 8634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:44.667018 8642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:44.667004 8639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:44.667145 8634 server_base.cc:1034] running on GCE node
W20260621 07:47:44.667054 8640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:44.667594 8634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:44.668175 8634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:44.669370 8634 hybrid_clock.cc:648] HybridClock initialized: now 1782028064669356 us; error 35 us; skew 500 ppm
Jun 21 07:47:44 dist-test-slave-lzx3 krb5kdc[8237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028064, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.192.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:44.671893 8634 init.cc:377] Logged in from keytab as oryx/127.0.192.195@KRBTEST.COM (short username oryx)
I20260621 07:47:44.672767 8634 webserver.cc:460] Webserver started at http://127.0.192.195:40847/ using document root <none> and password file <none>
I20260621 07:47:44.673269 8634 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:44.673372 8634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:44.673568 8634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:44.675412 8634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-2/data/instance:
uuid: "2c35196c6d7d417b936d34cd9d912d4e"
format_stamp: "Formatted at 2026-06-21 07:47:44 on dist-test-slave-lzx3"
server_key: "0954cb5bc529b9bd29cc97f0af34ba02"
server_key_iv: "916e6c0f45c90ca288cf2af6c4d907a1"
server_key_version: "encryptionkey@0"
I20260621 07:47:44.675930 8634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-2/wal/instance:
uuid: "2c35196c6d7d417b936d34cd9d912d4e"
format_stamp: "Formatted at 2026-06-21 07:47:44 on dist-test-slave-lzx3"
server_key: "0954cb5bc529b9bd29cc97f0af34ba02"
server_key_iv: "916e6c0f45c90ca288cf2af6c4d907a1"
server_key_version: "encryptionkey@0"
I20260621 07:47:44.679500 8634 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260621 07:47:44.681646 8649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:44.682700 8634 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 07:47:44.682824 8634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-2/wal
uuid: "2c35196c6d7d417b936d34cd9d912d4e"
format_stamp: "Formatted at 2026-06-21 07:47:44 on dist-test-slave-lzx3"
server_key: "0954cb5bc529b9bd29cc97f0af34ba02"
server_key_iv: "916e6c0f45c90ca288cf2af6c4d907a1"
server_key_version: "encryptionkey@0"
I20260621 07:47:44.682930 8634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:44.697083 8634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:44.700071 8634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:44.700251 8634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:44.701557 8634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:47:44.701639 8634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:44.701715 8634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:47:44.701761 8634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 07:47:44 dist-test-slave-lzx3 krb5kdc[8237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028064, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.192.195@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM
I20260621 07:47:44.712051 8634 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.195:44737
I20260621 07:47:44.712109 8764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.195:44737 every 8 connection(s)
I20260621 07:47:44.713032 8634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
I20260621 07:47:44.714743 8331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:44.702194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:56027 (local address 127.0.192.254:45399)
0621 07:47:44.702324 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:44.702329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:44.703179 (+ 850us) server_negotiation.cc:407] Connection header received
0621 07:47:44.704106 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:44.704110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:44.704163 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:44.704256 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:44.705549 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.706133 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.707292 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.707509 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.707616 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:44.711218 (+ 3602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:44.711222 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:44.711224 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:44.711255 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:44.712799 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:44.713335 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:44.713338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:44.713356 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:44.713450 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:44.713820 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:44.713823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:44.713825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:44.714031 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:44.714111 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:44.714515 (+ 404us) server_negotiation.cc:299] Negotiation successful
0621 07:47:44.714625 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 07:47:44.715507 8658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:44.702572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45399 (local address 127.0.192.195:56027)
0621 07:47:44.703008 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:44.703050 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:44.703921 (+ 871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:44.704523 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:44.704552 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:44.705009 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:44.705403 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:44.705415 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.706259 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.706263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:44.707183 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:44.707191 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.707646 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.707649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:44.707813 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:44.708411 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:44.708432 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:44.711045 (+ 2613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:44.712878 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:44.712883 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:44.712894 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:44.713240 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:44.713573 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:44.713576 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:44.713578 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:44.713717 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:44.714117 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:44.714122 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:44.714305 (+ 183us) client_negotiation.cc:769] Sending connection context
0621 07:47:44.714445 (+ 140us) client_negotiation.cc:240] Negotiation successful
0621 07:47:44.714693 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":113,"threads_started":1}
I20260621 07:47:44.716753 8765 heartbeater.cc:347] Connected to a master server at 127.0.192.254:45399
I20260621 07:47:44.716889 8765 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:44.717352 8765 heartbeater.cc:511] Master 127.0.192.254:45399 requested a full tablet report, sending...
I20260621 07:47:44.718307 8281 ts_manager.cc:194] Registered new tserver with Master: 2c35196c6d7d417b936d34cd9d912d4e (127.0.192.195:44737)
I20260621 07:47:44.718667 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 8634
I20260621 07:47:44.718791 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-2/wal/instance
I20260621 07:47:44.719020 8281 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.192.195@KRBTEST.COM'} at 127.0.192.195:56027
I20260621 07:47:44.719112 771 external_mini_cluster.cc:1428] Setting key 237ee171ef03939703e6bdda851e9028
I20260621 07:47:44.721175 771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:47:44.726052 8331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:44.719480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:51767 (local address 127.0.192.254:45399)
0621 07:47:44.719602 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:44.719606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:44.719619 (+ 13us) server_negotiation.cc:407] Connection header received
0621 07:47:44.719652 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:44.719654 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:44.719692 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:44.719767 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:44.720237 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.720939 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.721746 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.721968 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.722070 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:44.723187 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:44.723195 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:44.723197 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:44.723218 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:44.724496 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:44.724920 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:44.724925 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:44.724929 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:44.725005 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:44.725272 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:44.725274 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:44.725276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:44.725433 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:44.725532 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:44.725810 (+ 278us) server_negotiation.cc:299] Negotiation successful
0621 07:47:44.725907 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 07:47:44.726089 8771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:44.718692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45399 (local address 127.0.192.195:51767)
0621 07:47:44.718979 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:44.718994 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:44.719076 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:44.719794 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:44.719796 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:44.719993 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:44.720119 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:44.720126 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.721066 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.721068 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:44.721634 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:44.721642 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.722070 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.722074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:44.722166 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:44.722680 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:44.722698 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:44.723054 (+ 356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:44.724640 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:44.724643 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:44.724645 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:44.724826 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:44.725096 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:44.725098 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:44.725100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:44.725179 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:44.725535 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:44.725538 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:44.725658 (+ 120us) client_negotiation.cc:769] Sending connection context
0621 07:47:44.725761 (+ 103us) client_negotiation.cc:240] Negotiation successful
0621 07:47:44.725889 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":87,"threads_started":1}
Jun 21 07:47:44 dist-test-slave-lzx3 krb5kdc[8237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028064, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 07:47:44 dist-test-slave-lzx3 krb5kdc[8237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:47:44 dist-test-slave-lzx3 krb5kdc[8237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
W20260621 07:47:44.743775 8331 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:44.738139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46462 (local address 127.0.192.254:45399)
0621 07:47:44.738278 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:44.738281 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:44.738293 (+ 12us) server_negotiation.cc:407] Connection header received
0621 07:47:44.738327 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:44.738328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:44.738360 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:44.738431 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:44.739052 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.739531 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.740321 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.740508 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.740641 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:44.743616 (+ 2975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46462: BlockingRecv error: recv got EOF from 127.0.0.1:46462 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 21 07:47:44 dist-test-slave-lzx3 krb5kdc[8237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028064, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM
I20260621 07:47:44.751981 8331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:44.745085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46474 (local address 127.0.192.254:45399)
0621 07:47:44.745214 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:44.745217 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:44.745342 (+ 125us) server_negotiation.cc:407] Connection header received
0621 07:47:44.745433 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:44.745435 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:44.745477 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:44.745538 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:44.746035 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.746483 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.747201 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.747379 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.747443 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:44.749422 (+ 1979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:44.749425 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:44.749427 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:44.749446 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:44.750791 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:44.751142 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:44.751144 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:44.751145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:44.751209 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:44.751412 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:44.751413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:44.751415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:44.751561 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:44.751649 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:44.751809 (+ 160us) server_negotiation.cc:299] Negotiation successful
0621 07:47:44.751883 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 07:47:44.754402 8281 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:47:44.757215 8281 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:47:44.770433 8787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:44.764796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.195:44737 (local address 127.0.0.1:58036)
0621 07:47:44.765196 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:44.765209 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:44.765313 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:44.766684 (+ 1371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:44.766687 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:44.766703 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:44.766780 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:44.766787 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.767684 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.767687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:44.768873 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:44.768889 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.769933 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.769937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:44.770032 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:44.770069 (+ 37us) client_negotiation.cc:769] Sending connection context
0621 07:47:44.770180 (+ 111us) client_negotiation.cc:240] Negotiation successful
0621 07:47:44.770295 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":77,"spinlock_wait_cycles":2816,"thread_start_us":89,"threads_started":1}
I20260621 07:47:44.770449 8790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:44.764961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58036 (local address 127.0.192.195:44737)
0621 07:47:44.766348 (+ 1387us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:44.766359 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:44.766385 (+ 26us) server_negotiation.cc:407] Connection header received
0621 07:47:44.766444 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:44.766447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:44.766564 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:44.766704 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:44.766866 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.767565 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.768999 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.769801 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.769919 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:44.770073 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:44.770215 (+ 142us) server_negotiation.cc:299] Negotiation successful
0621 07:47:44.770312 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1292,"thread_start_us":101,"threads_started":1}
I20260621 07:47:44.770747 8792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:44.765632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45632 (local address 127.0.192.193:33397)
0621 07:47:44.765897 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:44.765908 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:44.765929 (+ 21us) server_negotiation.cc:407] Connection header received
0621 07:47:44.766017 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:44.766024 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:44.766202 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:44.766355 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:44.766720 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.767759 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.769116 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.769845 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.769931 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:44.770073 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:44.770509 (+ 436us) server_negotiation.cc:299] Negotiation successful
0621 07:47:44.770604 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":83,"threads_started":1}
I20260621 07:47:44.770877 8789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:44.764507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:43285 (local address 127.0.0.1:53012)
0621 07:47:44.765147 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:44.765191 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:44.765308 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:44.766077 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:44.766082 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:44.766102 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:44.766209 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:44.766216 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.767647 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.767653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:44.769444 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:44.769451 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.770495 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.770498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:44.770588 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:44.770608 (+ 20us) client_negotiation.cc:769] Sending connection context
0621 07:47:44.770675 (+ 67us) client_negotiation.cc:240] Negotiation successful
0621 07:47:44.770740 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"mutex_wait_us":672,"spinlock_wait_cycles":4992,"thread_start_us":133,"threads_started":1}
I20260621 07:47:44.771229 8788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:44.764637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53012 (local address 127.0.192.194:43285)
0621 07:47:44.765588 (+ 951us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:44.765600 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:44.765622 (+ 22us) server_negotiation.cc:407] Connection header received
0621 07:47:44.765690 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:44.765694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:44.765911 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:44.766078 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:44.766334 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.767528 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.769549 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.770197 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.770283 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:44.770432 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:44.771034 (+ 602us) server_negotiation.cc:299] Negotiation successful
0621 07:47:44.771123 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":854,"thread_start_us":130,"threads_started":1}
I20260621 07:47:44.772038 8791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:44.765239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:33397 (local address 127.0.0.1:45632)
0621 07:47:44.765464 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:44.765476 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:44.765548 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:44.766500 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:44.766503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:44.766519 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:44.766597 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:44.766602 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.767858 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.767864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:44.769000 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:44.769009 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.770135 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.770144 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:44.770255 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:44.770276 (+ 21us) client_negotiation.cc:769] Sending connection context
0621 07:47:44.770343 (+ 67us) client_negotiation.cc:240] Negotiation successful
0621 07:47:44.770402 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":7,"thread_start_us":83,"threads_started":1}
I20260621 07:47:44.773566 8559 tablet_service.cc:1467] Processing CreateTablet for tablet 41526bb7be5b4c06a692561aca86efba (DEFAULT_TABLE table=test-table [id=7accacc111914117a7dc2be8d3083753]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:47:44.773535 8699 tablet_service.cc:1467] Processing CreateTablet for tablet 41526bb7be5b4c06a692561aca86efba (DEFAULT_TABLE table=test-table [id=7accacc111914117a7dc2be8d3083753]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:47:44.774453 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41526bb7be5b4c06a692561aca86efba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:44.774469 8699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41526bb7be5b4c06a692561aca86efba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:44.774430 8413 tablet_service.cc:1467] Processing CreateTablet for tablet 41526bb7be5b4c06a692561aca86efba (DEFAULT_TABLE table=test-table [id=7accacc111914117a7dc2be8d3083753]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:47:44.775307 8413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41526bb7be5b4c06a692561aca86efba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:44.780978 8793 tablet_bootstrap.cc:492] T 41526bb7be5b4c06a692561aca86efba P 0f20308d9e324531a57e487bd730ba8f: Bootstrap starting.
I20260621 07:47:44.782222 8794 tablet_bootstrap.cc:492] T 41526bb7be5b4c06a692561aca86efba P 2c35196c6d7d417b936d34cd9d912d4e: Bootstrap starting.
I20260621 07:47:44.783283 8793 tablet_bootstrap.cc:654] T 41526bb7be5b4c06a692561aca86efba P 0f20308d9e324531a57e487bd730ba8f: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:44.783957 8793 log.cc:824] T 41526bb7be5b4c06a692561aca86efba P 0f20308d9e324531a57e487bd730ba8f: Log is configured to *not* fsync() on all Append() calls
I20260621 07:47:44.783998 8794 tablet_bootstrap.cc:654] T 41526bb7be5b4c06a692561aca86efba P 2c35196c6d7d417b936d34cd9d912d4e: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:44.784773 8794 log.cc:824] T 41526bb7be5b4c06a692561aca86efba P 2c35196c6d7d417b936d34cd9d912d4e: Log is configured to *not* fsync() on all Append() calls
I20260621 07:47:44.784893 8795 tablet_bootstrap.cc:492] T 41526bb7be5b4c06a692561aca86efba P 73148a30e9e54a29b9ab2894b5f3f8ae: Bootstrap starting.
I20260621 07:47:44.785889 8793 tablet_bootstrap.cc:492] T 41526bb7be5b4c06a692561aca86efba P 0f20308d9e324531a57e487bd730ba8f: No bootstrap required, opened a new log
I20260621 07:47:44.786068 8793 ts_tablet_manager.cc:1397] T 41526bb7be5b4c06a692561aca86efba P 0f20308d9e324531a57e487bd730ba8f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260621 07:47:44.786440 8794 tablet_bootstrap.cc:492] T 41526bb7be5b4c06a692561aca86efba P 2c35196c6d7d417b936d34cd9d912d4e: No bootstrap required, opened a new log
I20260621 07:47:44.786655 8794 ts_tablet_manager.cc:1397] T 41526bb7be5b4c06a692561aca86efba P 2c35196c6d7d417b936d34cd9d912d4e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260621 07:47:44.786862 8795 tablet_bootstrap.cc:654] T 41526bb7be5b4c06a692561aca86efba P 73148a30e9e54a29b9ab2894b5f3f8ae: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:44.787851 8795 log.cc:824] T 41526bb7be5b4c06a692561aca86efba P 73148a30e9e54a29b9ab2894b5f3f8ae: Log is configured to *not* fsync() on all Append() calls
I20260621 07:47:44.789660 8795 tablet_bootstrap.cc:492] T 41526bb7be5b4c06a692561aca86efba P 73148a30e9e54a29b9ab2894b5f3f8ae: No bootstrap required, opened a new log
I20260621 07:47:44.789824 8795 ts_tablet_manager.cc:1397] T 41526bb7be5b4c06a692561aca86efba P 73148a30e9e54a29b9ab2894b5f3f8ae: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260621 07:47:44.790076 8793 raft_consensus.cc:348] T 41526bb7be5b4c06a692561aca86efba P 0f20308d9e324531a57e487bd730ba8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } }
I20260621 07:47:44.790158 8794 raft_consensus.cc:348] T 41526bb7be5b4c06a692561aca86efba P 2c35196c6d7d417b936d34cd9d912d4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } }
I20260621 07:47:44.790313 8793 raft_consensus.cc:374] T 41526bb7be5b4c06a692561aca86efba P 0f20308d9e324531a57e487bd730ba8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:44.790359 8793 raft_consensus.cc:729] T 41526bb7be5b4c06a692561aca86efba P 0f20308d9e324531a57e487bd730ba8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f20308d9e324531a57e487bd730ba8f, State: Initialized, Role: FOLLOWER
I20260621 07:47:44.790364 8794 raft_consensus.cc:374] T 41526bb7be5b4c06a692561aca86efba P 2c35196c6d7d417b936d34cd9d912d4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:44.790416 8794 raft_consensus.cc:729] T 41526bb7be5b4c06a692561aca86efba P 2c35196c6d7d417b936d34cd9d912d4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c35196c6d7d417b936d34cd9d912d4e, State: Initialized, Role: FOLLOWER
I20260621 07:47:44.790747 8793 consensus_queue.cc:260] T 41526bb7be5b4c06a692561aca86efba P 0f20308d9e324531a57e487bd730ba8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } }
I20260621 07:47:44.790776 8794 consensus_queue.cc:260] T 41526bb7be5b4c06a692561aca86efba P 2c35196c6d7d417b936d34cd9d912d4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } }
I20260621 07:47:44.791375 8625 heartbeater.cc:503] Master 127.0.192.254:45399 was elected leader, sending a full tablet report...
I20260621 07:47:44.791704 8793 ts_tablet_manager.cc:1428] T 41526bb7be5b4c06a692561aca86efba P 0f20308d9e324531a57e487bd730ba8f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260621 07:47:44.792616 8794 ts_tablet_manager.cc:1428] T 41526bb7be5b4c06a692561aca86efba P 2c35196c6d7d417b936d34cd9d912d4e: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260621 07:47:44.792932 8765 heartbeater.cc:503] Master 127.0.192.254:45399 was elected leader, sending a full tablet report...
I20260621 07:47:44.793313 8795 raft_consensus.cc:348] T 41526bb7be5b4c06a692561aca86efba P 73148a30e9e54a29b9ab2894b5f3f8ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } }
I20260621 07:47:44.793557 8795 raft_consensus.cc:374] T 41526bb7be5b4c06a692561aca86efba P 73148a30e9e54a29b9ab2894b5f3f8ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:44.793602 8795 raft_consensus.cc:729] T 41526bb7be5b4c06a692561aca86efba P 73148a30e9e54a29b9ab2894b5f3f8ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73148a30e9e54a29b9ab2894b5f3f8ae, State: Initialized, Role: FOLLOWER
I20260621 07:47:44.793941 8795 consensus_queue.cc:260] T 41526bb7be5b4c06a692561aca86efba P 73148a30e9e54a29b9ab2894b5f3f8ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } }
I20260621 07:47:44.794464 8485 heartbeater.cc:503] Master 127.0.192.254:45399 was elected leader, sending a full tablet report...
I20260621 07:47:44.796080 8795 ts_tablet_manager.cc:1428] T 41526bb7be5b4c06a692561aca86efba P 73148a30e9e54a29b9ab2894b5f3f8ae: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260621 07:47:44.805672 8799 raft_consensus.cc:482] T 41526bb7be5b4c06a692561aca86efba P 0f20308d9e324531a57e487bd730ba8f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:47:44.805830 8799 raft_consensus.cc:504] T 41526bb7be5b4c06a692561aca86efba P 0f20308d9e324531a57e487bd730ba8f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } }
I20260621 07:47:44.806699 8799 leader_election.cc:288] T 41526bb7be5b4c06a692561aca86efba P 0f20308d9e324531a57e487bd730ba8f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73148a30e9e54a29b9ab2894b5f3f8ae (127.0.192.193:33397), 2c35196c6d7d417b936d34cd9d912d4e (127.0.192.195:44737)
W20260621 07:47:44.810348 8486 tablet.cc:2369] T 41526bb7be5b4c06a692561aca86efba P 73148a30e9e54a29b9ab2894b5f3f8ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:47:44.811295 8518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:44.806943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.195:44737 (local address 127.0.192.194:46143)
0621 07:47:44.807121 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:44.807136 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:44.807212 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:44.807490 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:44.807492 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:44.807503 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:44.807625 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:44.807631 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.808697 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.808700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:44.810037 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:44.810047 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.810847 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.810850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:44.810951 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:44.810977 (+ 26us) client_negotiation.cc:769] Sending connection context
0621 07:47:44.811039 (+ 62us) client_negotiation.cc:240] Negotiation successful
0621 07:47:44.811142 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":30}
I20260621 07:47:44.811276 8790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:44.807047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:46143 (local address 127.0.192.195:44737)
0621 07:47:44.807170 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:44.807174 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:44.807188 (+ 14us) server_negotiation.cc:407] Connection header received
0621 07:47:44.807319 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:44.807321 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:44.807381 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:44.807466 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:44.807727 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.808560 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.810143 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.810704 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.810828 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:44.810873 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:44.811080 (+ 207us) server_negotiation.cc:299] Negotiation successful
0621 07:47:44.811156 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 07:47:44.811570 8792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:44.807035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:48403 (local address 127.0.192.193:33397)
0621 07:47:44.807130 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:44.807136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:44.807373 (+ 237us) server_negotiation.cc:407] Connection header received
0621 07:47:44.807443 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:44.807448 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:44.807503 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:44.807572 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:44.807819 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.808618 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.810392 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.810994 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.811134 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:44.811174 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:44.811382 (+ 208us) server_negotiation.cc:299] Negotiation successful
0621 07:47:44.811468 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260621 07:47:44.811870 8802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:44.806943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:33397 (local address 127.0.192.194:48403)
0621 07:47:44.807275 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:44.807289 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:44.807370 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:44.807596 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:44.807599 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:44.807611 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:44.807720 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:44.807726 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.808705 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.808708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:44.810295 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:44.810306 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.811133 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.811136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:44.811220 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:44.811240 (+ 20us) client_negotiation.cc:769] Sending connection context
0621 07:47:44.811314 (+ 74us) client_negotiation.cc:240] Negotiation successful
0621 07:47:44.811741 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":77,"thread_start_us":88,"threads_started":1}
I20260621 07:47:44.811846 8719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41526bb7be5b4c06a692561aca86efba" candidate_uuid: "0f20308d9e324531a57e487bd730ba8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c35196c6d7d417b936d34cd9d912d4e" is_pre_election: true
I20260621 07:47:44.812116 8719 raft_consensus.cc:2454] T 41526bb7be5b4c06a692561aca86efba P 2c35196c6d7d417b936d34cd9d912d4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f20308d9e324531a57e487bd730ba8f in term 0.
I20260621 07:47:44.812428 8439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41526bb7be5b4c06a692561aca86efba" candidate_uuid: "0f20308d9e324531a57e487bd730ba8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" is_pre_election: true
I20260621 07:47:44.812507 8511 leader_election.cc:302] T 41526bb7be5b4c06a692561aca86efba P 0f20308d9e324531a57e487bd730ba8f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f20308d9e324531a57e487bd730ba8f, 2c35196c6d7d417b936d34cd9d912d4e; no voters:
I20260621 07:47:44.812706 8439 raft_consensus.cc:2454] T 41526bb7be5b4c06a692561aca86efba P 73148a30e9e54a29b9ab2894b5f3f8ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f20308d9e324531a57e487bd730ba8f in term 0.
I20260621 07:47:44.812824 8799 raft_consensus.cc:2781] T 41526bb7be5b4c06a692561aca86efba P 0f20308d9e324531a57e487bd730ba8f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:47:44.812929 8799 raft_consensus.cc:482] T 41526bb7be5b4c06a692561aca86efba P 0f20308d9e324531a57e487bd730ba8f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:47:44.812988 8799 raft_consensus.cc:3037] T 41526bb7be5b4c06a692561aca86efba P 0f20308d9e324531a57e487bd730ba8f [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:44.813882 8799 raft_consensus.cc:504] T 41526bb7be5b4c06a692561aca86efba P 0f20308d9e324531a57e487bd730ba8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } }
I20260621 07:47:44.814198 8799 leader_election.cc:288] T 41526bb7be5b4c06a692561aca86efba P 0f20308d9e324531a57e487bd730ba8f [CANDIDATE]: Term 1 election: Requested vote from peers 73148a30e9e54a29b9ab2894b5f3f8ae (127.0.192.193:33397), 2c35196c6d7d417b936d34cd9d912d4e (127.0.192.195:44737)
I20260621 07:47:44.814468 8719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41526bb7be5b4c06a692561aca86efba" candidate_uuid: "0f20308d9e324531a57e487bd730ba8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c35196c6d7d417b936d34cd9d912d4e"
I20260621 07:47:44.814468 8439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41526bb7be5b4c06a692561aca86efba" candidate_uuid: "0f20308d9e324531a57e487bd730ba8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae"
I20260621 07:47:44.814567 8719 raft_consensus.cc:3037] T 41526bb7be5b4c06a692561aca86efba P 2c35196c6d7d417b936d34cd9d912d4e [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:44.814567 8439 raft_consensus.cc:3037] T 41526bb7be5b4c06a692561aca86efba P 73148a30e9e54a29b9ab2894b5f3f8ae [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:44.815744 8719 raft_consensus.cc:2454] T 41526bb7be5b4c06a692561aca86efba P 2c35196c6d7d417b936d34cd9d912d4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f20308d9e324531a57e487bd730ba8f in term 1.
I20260621 07:47:44.815744 8439 raft_consensus.cc:2454] T 41526bb7be5b4c06a692561aca86efba P 73148a30e9e54a29b9ab2894b5f3f8ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f20308d9e324531a57e487bd730ba8f in term 1.
I20260621 07:47:44.816008 8511 leader_election.cc:302] T 41526bb7be5b4c06a692561aca86efba P 0f20308d9e324531a57e487bd730ba8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f20308d9e324531a57e487bd730ba8f, 2c35196c6d7d417b936d34cd9d912d4e; no voters:
I20260621 07:47:44.816201 8799 raft_consensus.cc:2781] T 41526bb7be5b4c06a692561aca86efba P 0f20308d9e324531a57e487bd730ba8f [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:47:44.816366 8799 raft_consensus.cc:686] T 41526bb7be5b4c06a692561aca86efba P 0f20308d9e324531a57e487bd730ba8f [term 1 LEADER]: Becoming Leader. State: Replica: 0f20308d9e324531a57e487bd730ba8f, State: Running, Role: LEADER
I20260621 07:47:44.816686 8799 consensus_queue.cc:237] T 41526bb7be5b4c06a692561aca86efba P 0f20308d9e324531a57e487bd730ba8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } }
I20260621 07:47:44.819717 8282 catalog_manager.cc:5495] T 41526bb7be5b4c06a692561aca86efba P 0f20308d9e324531a57e487bd730ba8f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f20308d9e324531a57e487bd730ba8f (127.0.192.194). New cstate: current_term: 1 leader_uuid: "0f20308d9e324531a57e487bd730ba8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } health_report { overall_health: UNKNOWN } } }
I20260621 07:47:44.839186 8788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:44.835779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53016 (local address 127.0.192.194:43285)
0621 07:47:44.835904 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:44.835910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:44.835925 (+ 15us) server_negotiation.cc:407] Connection header received
0621 07:47:44.836116 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:44.836118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:44.836164 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:44.836268 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:47:44.836519 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.837018 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.837821 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:44.838043 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:44.838146 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:44.838432 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:47:44.838804 (+ 372us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:47:44.838925 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:44.839005 (+ 80us) server_negotiation.cc:299] Negotiation successful
0621 07:47:44.839052 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 07:47:44.844219 8439 raft_consensus.cc:1261] T 41526bb7be5b4c06a692561aca86efba P 73148a30e9e54a29b9ab2894b5f3f8ae [term 1 FOLLOWER]: Refusing update from remote peer 0f20308d9e324531a57e487bd730ba8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:47:44.844576 8719 raft_consensus.cc:1261] T 41526bb7be5b4c06a692561aca86efba P 2c35196c6d7d417b936d34cd9d912d4e [term 1 FOLLOWER]: Refusing update from remote peer 0f20308d9e324531a57e487bd730ba8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:47:44.844874 8799 consensus_queue.cc:1035] T 41526bb7be5b4c06a692561aca86efba P 0f20308d9e324531a57e487bd730ba8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:47:44.845180 8803 consensus_queue.cc:1035] T 41526bb7be5b4c06a692561aca86efba P 0f20308d9e324531a57e487bd730ba8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:47:44.850572 8806 mvcc.cc:205] Tried to move back new op lower bound from 7299186953595588608 to 7299186953491476480. Current Snapshot: MvccSnapshot[applied={T|T < 7299186953595588608}]
I20260621 07:47:44.851267 8807 mvcc.cc:205] Tried to move back new op lower bound from 7299186953595588608 to 7299186953491476480. Current Snapshot: MvccSnapshot[applied={T|T < 7299186953595588608}]
I20260621 07:47:44.852831 8808 mvcc.cc:205] Tried to move back new op lower bound from 7299186953595588608 to 7299186953491476480. Current Snapshot: MvccSnapshot[applied={T|T < 7299186953595588608}]
I20260621 07:47:44.858659 8282 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:47:44.858811 8282 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:47:44.860973 8282 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a5a7e6a80b74aebbeb361add74240ab: Sending DeleteTablet for 3 replicas of tablet 41526bb7be5b4c06a692561aca86efba
I20260621 07:47:44.861737 8699 tablet_service.cc:1514] Processing DeleteTablet for tablet 41526bb7be5b4c06a692561aca86efba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:47:44 UTC) from {username='oryx', principal='oryx/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:58036
I20260621 07:47:44.861773 8413 tablet_service.cc:1514] Processing DeleteTablet for tablet 41526bb7be5b4c06a692561aca86efba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:47:44 UTC) from {username='oryx', principal='oryx/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:45632
I20260621 07:47:44.862196 8817 tablet_replica.cc:330] T 41526bb7be5b4c06a692561aca86efba P 2c35196c6d7d417b936d34cd9d912d4e: stopping tablet replica
I20260621 07:47:44.862197 8559 tablet_service.cc:1514] Processing DeleteTablet for tablet 41526bb7be5b4c06a692561aca86efba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:47:44 UTC) from {username='oryx', principal='oryx/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:53012
I20260621 07:47:44.862459 8816 tablet_replica.cc:330] T 41526bb7be5b4c06a692561aca86efba P 73148a30e9e54a29b9ab2894b5f3f8ae: stopping tablet replica
I20260621 07:47:44.862670 8818 tablet_replica.cc:330] T 41526bb7be5b4c06a692561aca86efba P 0f20308d9e324531a57e487bd730ba8f: stopping tablet replica
I20260621 07:47:44.862874 8818 raft_consensus.cc:2229] T 41526bb7be5b4c06a692561aca86efba P 0f20308d9e324531a57e487bd730ba8f [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:47:44.863004 8816 raft_consensus.cc:2229] T 41526bb7be5b4c06a692561aca86efba P 73148a30e9e54a29b9ab2894b5f3f8ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:47:44.862946 8282 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260621 07:47:44.863211 8818 raft_consensus.cc:2258] T 41526bb7be5b4c06a692561aca86efba P 0f20308d9e324531a57e487bd730ba8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:47:44.863272 8816 raft_consensus.cc:2258] T 41526bb7be5b4c06a692561aca86efba P 73148a30e9e54a29b9ab2894b5f3f8ae [term 1 FOLLOWER]: Raft consensus is shut down!
W20260621 07:47:44.863359 8282 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:47:44.862805 8817 raft_consensus.cc:2229] T 41526bb7be5b4c06a692561aca86efba P 2c35196c6d7d417b936d34cd9d912d4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:47:44.863962 8817 raft_consensus.cc:2258] T 41526bb7be5b4c06a692561aca86efba P 2c35196c6d7d417b936d34cd9d912d4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:47:44.864501 8818 ts_tablet_manager.cc:1905] T 41526bb7be5b4c06a692561aca86efba P 0f20308d9e324531a57e487bd730ba8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:47:44.866873 8816 ts_tablet_manager.cc:1905] T 41526bb7be5b4c06a692561aca86efba P 73148a30e9e54a29b9ab2894b5f3f8ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:47:44.867422 8817 ts_tablet_manager.cc:1905] T 41526bb7be5b4c06a692561aca86efba P 2c35196c6d7d417b936d34cd9d912d4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:47:44.868364 8818 ts_tablet_manager.cc:1918] T 41526bb7be5b4c06a692561aca86efba P 0f20308d9e324531a57e487bd730ba8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:47:44.868465 8818 log.cc:1196] T 41526bb7be5b4c06a692561aca86efba P 0f20308d9e324531a57e487bd730ba8f: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-1/wal/wals/41526bb7be5b4c06a692561aca86efba
I20260621 07:47:44.868827 8413 tablet_service.cc:1467] Processing CreateTablet for tablet c2182cf9de6e469ead99160fcc31dabe (DEFAULT_TABLE table=test-table [id=30505adcfd874a01896b894f436662bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:47:44.869105 8413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2182cf9de6e469ead99160fcc31dabe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:44.869647 8816 ts_tablet_manager.cc:1918] T 41526bb7be5b4c06a692561aca86efba P 73148a30e9e54a29b9ab2894b5f3f8ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:47:44.869760 8816 log.cc:1196] T 41526bb7be5b4c06a692561aca86efba P 73148a30e9e54a29b9ab2894b5f3f8ae: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-0/wal/wals/41526bb7be5b4c06a692561aca86efba
I20260621 07:47:44.870085 8816 ts_tablet_manager.cc:1939] T 41526bb7be5b4c06a692561aca86efba P 73148a30e9e54a29b9ab2894b5f3f8ae: Deleting consensus metadata
I20260621 07:47:44.870235 8817 ts_tablet_manager.cc:1918] T 41526bb7be5b4c06a692561aca86efba P 2c35196c6d7d417b936d34cd9d912d4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:47:44.870360 8817 log.cc:1196] T 41526bb7be5b4c06a692561aca86efba P 2c35196c6d7d417b936d34cd9d912d4e: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-2/wal/wals/41526bb7be5b4c06a692561aca86efba
I20260621 07:47:44.870675 8817 ts_tablet_manager.cc:1939] T 41526bb7be5b4c06a692561aca86efba P 2c35196c6d7d417b936d34cd9d912d4e: Deleting consensus metadata
I20260621 07:47:44.871341 8559 tablet_service.cc:1467] Processing CreateTablet for tablet c2182cf9de6e469ead99160fcc31dabe (DEFAULT_TABLE table=test-table [id=30505adcfd874a01896b894f436662bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:47:44.871438 8818 ts_tablet_manager.cc:1939] T 41526bb7be5b4c06a692561aca86efba P 0f20308d9e324531a57e487bd730ba8f: Deleting consensus metadata
I20260621 07:47:44.871606 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2182cf9de6e469ead99160fcc31dabe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:44.872512 8266 catalog_manager.cc:4841] TS 73148a30e9e54a29b9ab2894b5f3f8ae (127.0.192.193:33397): tablet 41526bb7be5b4c06a692561aca86efba (table test-table [id=7accacc111914117a7dc2be8d3083753]) successfully deleted
I20260621 07:47:44.872627 8795 tablet_bootstrap.cc:492] T c2182cf9de6e469ead99160fcc31dabe P 73148a30e9e54a29b9ab2894b5f3f8ae: Bootstrap starting.
I20260621 07:47:44.872741 8266 catalog_manager.cc:4841] TS 0f20308d9e324531a57e487bd730ba8f (127.0.192.194:43285): tablet 41526bb7be5b4c06a692561aca86efba (table test-table [id=7accacc111914117a7dc2be8d3083753]) successfully deleted
I20260621 07:47:44.873526 8793 tablet_bootstrap.cc:492] T c2182cf9de6e469ead99160fcc31dabe P 0f20308d9e324531a57e487bd730ba8f: Bootstrap starting.
I20260621 07:47:44.873535 8795 tablet_bootstrap.cc:654] T c2182cf9de6e469ead99160fcc31dabe P 73148a30e9e54a29b9ab2894b5f3f8ae: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:44.874276 8793 tablet_bootstrap.cc:654] T c2182cf9de6e469ead99160fcc31dabe P 0f20308d9e324531a57e487bd730ba8f: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:44.874641 8267 catalog_manager.cc:4841] TS 2c35196c6d7d417b936d34cd9d912d4e (127.0.192.195:44737): tablet 41526bb7be5b4c06a692561aca86efba (table test-table [id=7accacc111914117a7dc2be8d3083753]) successfully deleted
I20260621 07:47:44.875120 8795 tablet_bootstrap.cc:492] T c2182cf9de6e469ead99160fcc31dabe P 73148a30e9e54a29b9ab2894b5f3f8ae: No bootstrap required, opened a new log
I20260621 07:47:44.875133 8699 tablet_service.cc:1467] Processing CreateTablet for tablet c2182cf9de6e469ead99160fcc31dabe (DEFAULT_TABLE table=test-table [id=30505adcfd874a01896b894f436662bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:47:44.875236 8795 ts_tablet_manager.cc:1397] T c2182cf9de6e469ead99160fcc31dabe P 73148a30e9e54a29b9ab2894b5f3f8ae: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260621 07:47:44.875386 8699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2182cf9de6e469ead99160fcc31dabe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:44.875698 8795 raft_consensus.cc:348] T c2182cf9de6e469ead99160fcc31dabe P 73148a30e9e54a29b9ab2894b5f3f8ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } }
I20260621 07:47:44.875808 8795 raft_consensus.cc:374] T c2182cf9de6e469ead99160fcc31dabe P 73148a30e9e54a29b9ab2894b5f3f8ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:44.875842 8795 raft_consensus.cc:729] T c2182cf9de6e469ead99160fcc31dabe P 73148a30e9e54a29b9ab2894b5f3f8ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73148a30e9e54a29b9ab2894b5f3f8ae, State: Initialized, Role: FOLLOWER
I20260621 07:47:44.875953 8795 consensus_queue.cc:260] T c2182cf9de6e469ead99160fcc31dabe P 73148a30e9e54a29b9ab2894b5f3f8ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } }
I20260621 07:47:44.876266 8795 ts_tablet_manager.cc:1428] T c2182cf9de6e469ead99160fcc31dabe P 73148a30e9e54a29b9ab2894b5f3f8ae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260621 07:47:44.876783 8793 tablet_bootstrap.cc:492] T c2182cf9de6e469ead99160fcc31dabe P 0f20308d9e324531a57e487bd730ba8f: No bootstrap required, opened a new log
I20260621 07:47:44.876853 8793 ts_tablet_manager.cc:1397] T c2182cf9de6e469ead99160fcc31dabe P 0f20308d9e324531a57e487bd730ba8f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260621 07:47:44.877247 8793 raft_consensus.cc:348] T c2182cf9de6e469ead99160fcc31dabe P 0f20308d9e324531a57e487bd730ba8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } }
I20260621 07:47:44.877336 8793 raft_consensus.cc:374] T c2182cf9de6e469ead99160fcc31dabe P 0f20308d9e324531a57e487bd730ba8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:44.877363 8793 raft_consensus.cc:729] T c2182cf9de6e469ead99160fcc31dabe P 0f20308d9e324531a57e487bd730ba8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f20308d9e324531a57e487bd730ba8f, State: Initialized, Role: FOLLOWER
I20260621 07:47:44.877508 8794 tablet_bootstrap.cc:492] T c2182cf9de6e469ead99160fcc31dabe P 2c35196c6d7d417b936d34cd9d912d4e: Bootstrap starting.
I20260621 07:47:44.877465 8793 consensus_queue.cc:260] T c2182cf9de6e469ead99160fcc31dabe P 0f20308d9e324531a57e487bd730ba8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } }
I20260621 07:47:44.877704 8793 ts_tablet_manager.cc:1428] T c2182cf9de6e469ead99160fcc31dabe P 0f20308d9e324531a57e487bd730ba8f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260621 07:47:44.878536 8794 tablet_bootstrap.cc:654] T c2182cf9de6e469ead99160fcc31dabe P 2c35196c6d7d417b936d34cd9d912d4e: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:44.879562 8794 tablet_bootstrap.cc:492] T c2182cf9de6e469ead99160fcc31dabe P 2c35196c6d7d417b936d34cd9d912d4e: No bootstrap required, opened a new log
I20260621 07:47:44.879640 8794 ts_tablet_manager.cc:1397] T c2182cf9de6e469ead99160fcc31dabe P 2c35196c6d7d417b936d34cd9d912d4e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 07:47:44.880040 8794 raft_consensus.cc:348] T c2182cf9de6e469ead99160fcc31dabe P 2c35196c6d7d417b936d34cd9d912d4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } }
I20260621 07:47:44.880133 8794 raft_consensus.cc:374] T c2182cf9de6e469ead99160fcc31dabe P 2c35196c6d7d417b936d34cd9d912d4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:44.880160 8794 raft_consensus.cc:729] T c2182cf9de6e469ead99160fcc31dabe P 2c35196c6d7d417b936d34cd9d912d4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c35196c6d7d417b936d34cd9d912d4e, State: Initialized, Role: FOLLOWER
I20260621 07:47:44.880262 8794 consensus_queue.cc:260] T c2182cf9de6e469ead99160fcc31dabe P 2c35196c6d7d417b936d34cd9d912d4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } }
I20260621 07:47:44.880455 8794 ts_tablet_manager.cc:1428] T c2182cf9de6e469ead99160fcc31dabe P 2c35196c6d7d417b936d34cd9d912d4e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260621 07:47:44.963886 8766 tablet.cc:2369] T c2182cf9de6e469ead99160fcc31dabe P 2c35196c6d7d417b936d34cd9d912d4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:47:45.012058 8813 raft_consensus.cc:482] T c2182cf9de6e469ead99160fcc31dabe P 0f20308d9e324531a57e487bd730ba8f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:47:45.012192 8813 raft_consensus.cc:504] T c2182cf9de6e469ead99160fcc31dabe P 0f20308d9e324531a57e487bd730ba8f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } }
I20260621 07:47:45.012573 8813 leader_election.cc:288] T c2182cf9de6e469ead99160fcc31dabe P 0f20308d9e324531a57e487bd730ba8f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73148a30e9e54a29b9ab2894b5f3f8ae (127.0.192.193:33397), 2c35196c6d7d417b936d34cd9d912d4e (127.0.192.195:44737)
W20260621 07:47:45.012598 8626 tablet.cc:2369] T c2182cf9de6e469ead99160fcc31dabe P 0f20308d9e324531a57e487bd730ba8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:47:45.012960 8439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2182cf9de6e469ead99160fcc31dabe" candidate_uuid: "0f20308d9e324531a57e487bd730ba8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" is_pre_election: true
I20260621 07:47:45.013019 8719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2182cf9de6e469ead99160fcc31dabe" candidate_uuid: "0f20308d9e324531a57e487bd730ba8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c35196c6d7d417b936d34cd9d912d4e" is_pre_election: true
I20260621 07:47:45.013103 8439 raft_consensus.cc:2454] T c2182cf9de6e469ead99160fcc31dabe P 73148a30e9e54a29b9ab2894b5f3f8ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f20308d9e324531a57e487bd730ba8f in term 0.
I20260621 07:47:45.013147 8719 raft_consensus.cc:2454] T c2182cf9de6e469ead99160fcc31dabe P 2c35196c6d7d417b936d34cd9d912d4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f20308d9e324531a57e487bd730ba8f in term 0.
I20260621 07:47:45.013417 8510 leader_election.cc:302] T c2182cf9de6e469ead99160fcc31dabe P 0f20308d9e324531a57e487bd730ba8f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f20308d9e324531a57e487bd730ba8f, 73148a30e9e54a29b9ab2894b5f3f8ae; no voters:
I20260621 07:47:45.013589 8813 raft_consensus.cc:2781] T c2182cf9de6e469ead99160fcc31dabe P 0f20308d9e324531a57e487bd730ba8f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:47:45.013635 8813 raft_consensus.cc:482] T c2182cf9de6e469ead99160fcc31dabe P 0f20308d9e324531a57e487bd730ba8f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:47:45.013666 8813 raft_consensus.cc:3037] T c2182cf9de6e469ead99160fcc31dabe P 0f20308d9e324531a57e487bd730ba8f [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:45.014668 8813 raft_consensus.cc:504] T c2182cf9de6e469ead99160fcc31dabe P 0f20308d9e324531a57e487bd730ba8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } }
I20260621 07:47:45.014986 8813 leader_election.cc:288] T c2182cf9de6e469ead99160fcc31dabe P 0f20308d9e324531a57e487bd730ba8f [CANDIDATE]: Term 1 election: Requested vote from peers 73148a30e9e54a29b9ab2894b5f3f8ae (127.0.192.193:33397), 2c35196c6d7d417b936d34cd9d912d4e (127.0.192.195:44737)
I20260621 07:47:45.015316 8439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2182cf9de6e469ead99160fcc31dabe" candidate_uuid: "0f20308d9e324531a57e487bd730ba8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae"
I20260621 07:47:45.015435 8439 raft_consensus.cc:3037] T c2182cf9de6e469ead99160fcc31dabe P 73148a30e9e54a29b9ab2894b5f3f8ae [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:45.016559 8439 raft_consensus.cc:2454] T c2182cf9de6e469ead99160fcc31dabe P 73148a30e9e54a29b9ab2894b5f3f8ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f20308d9e324531a57e487bd730ba8f in term 1.
I20260621 07:47:45.016741 8719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2182cf9de6e469ead99160fcc31dabe" candidate_uuid: "0f20308d9e324531a57e487bd730ba8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c35196c6d7d417b936d34cd9d912d4e"
I20260621 07:47:45.016834 8510 leader_election.cc:302] T c2182cf9de6e469ead99160fcc31dabe P 0f20308d9e324531a57e487bd730ba8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f20308d9e324531a57e487bd730ba8f, 73148a30e9e54a29b9ab2894b5f3f8ae; no voters:
I20260621 07:47:45.016870 8719 raft_consensus.cc:3037] T c2182cf9de6e469ead99160fcc31dabe P 2c35196c6d7d417b936d34cd9d912d4e [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:45.016999 8813 raft_consensus.cc:2781] T c2182cf9de6e469ead99160fcc31dabe P 0f20308d9e324531a57e487bd730ba8f [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:47:45.017081 8813 raft_consensus.cc:686] T c2182cf9de6e469ead99160fcc31dabe P 0f20308d9e324531a57e487bd730ba8f [term 1 LEADER]: Becoming Leader. State: Replica: 0f20308d9e324531a57e487bd730ba8f, State: Running, Role: LEADER
I20260621 07:47:45.017220 8813 consensus_queue.cc:237] T c2182cf9de6e469ead99160fcc31dabe P 0f20308d9e324531a57e487bd730ba8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } }
I20260621 07:47:45.017939 8719 raft_consensus.cc:2454] T c2182cf9de6e469ead99160fcc31dabe P 2c35196c6d7d417b936d34cd9d912d4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f20308d9e324531a57e487bd730ba8f in term 1.
I20260621 07:47:45.018800 8281 catalog_manager.cc:5495] T c2182cf9de6e469ead99160fcc31dabe P 0f20308d9e324531a57e487bd730ba8f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f20308d9e324531a57e487bd730ba8f (127.0.192.194). New cstate: current_term: 1 leader_uuid: "0f20308d9e324531a57e487bd730ba8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } health_report { overall_health: UNKNOWN } } }
Jun 21 07:47:45 dist-test-slave-lzx3 krb5kdc[8237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028063, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.192.254@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM
I20260621 07:47:45.053931 8828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:45.045896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45399 (local address 127.0.0.1:46482)
0621 07:47:45.046163 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:45.046192 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:45.046286 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:45.046519 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:45.046523 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:45.047053 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:45.047192 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:45.047203 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:45.048055 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:45.048057 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:45.048682 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:45.048691 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:45.049072 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:45.049074 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:45.049136 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:45.049646 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:45.049657 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:45.050941 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:45.052608 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:45.052616 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:45.052629 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:45.052834 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:45.053067 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:45.053069 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:45.053070 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:45.053144 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:45.053458 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:45.053463 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:45.053555 (+ 92us) client_negotiation.cc:769] Sending connection context
0621 07:47:45.053673 (+ 118us) client_negotiation.cc:240] Negotiation successful
0621 07:47:45.053780 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":109,"threads_started":1}
I20260621 07:47:45.053946 8331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:45.045972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46482 (local address 127.0.192.254:45399)
0621 07:47:45.046082 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:45.046085 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:45.046260 (+ 175us) server_negotiation.cc:407] Connection header received
0621 07:47:45.046360 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:45.046361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:45.046394 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:45.046460 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:45.047308 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:45.047958 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:45.048785 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:45.048980 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:45.049052 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:45.051059 (+ 2007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:45.051062 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:45.051063 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:45.051082 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:45.052473 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:45.052917 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:45.052919 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:45.052920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:45.052984 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:45.053214 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:45.053216 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:45.053217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:45.053350 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:45.053416 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:45.053721 (+ 305us) server_negotiation.cc:299] Negotiation successful
0621 07:47:45.053829 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":43}
I20260621 07:47:45.057519 8281 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:46482:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260621 07:47:45.058140 8281 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:47:45.062569 8559 tablet_service.cc:1467] Processing CreateTablet for tablet 5fa6a43ca766412fbd638c20ea041949 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=baa1dc16b7e34888962777d0f7e74439]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 07:47:45.062568 8413 tablet_service.cc:1467] Processing CreateTablet for tablet 5fa6a43ca766412fbd638c20ea041949 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=baa1dc16b7e34888962777d0f7e74439]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 07:47:45.062774 8699 tablet_service.cc:1467] Processing CreateTablet for tablet 5fa6a43ca766412fbd638c20ea041949 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=baa1dc16b7e34888962777d0f7e74439]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 07:47:45.062898 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fa6a43ca766412fbd638c20ea041949. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:45.062898 8413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fa6a43ca766412fbd638c20ea041949. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:45.063117 8699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fa6a43ca766412fbd638c20ea041949. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:45.065455 8794 tablet_bootstrap.cc:492] T 5fa6a43ca766412fbd638c20ea041949 P 2c35196c6d7d417b936d34cd9d912d4e: Bootstrap starting.
I20260621 07:47:45.065456 8793 tablet_bootstrap.cc:492] T 5fa6a43ca766412fbd638c20ea041949 P 0f20308d9e324531a57e487bd730ba8f: Bootstrap starting.
I20260621 07:47:45.065495 8795 tablet_bootstrap.cc:492] T 5fa6a43ca766412fbd638c20ea041949 P 73148a30e9e54a29b9ab2894b5f3f8ae: Bootstrap starting.
I20260621 07:47:45.066220 8794 tablet_bootstrap.cc:654] T 5fa6a43ca766412fbd638c20ea041949 P 2c35196c6d7d417b936d34cd9d912d4e: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:45.066265 8795 tablet_bootstrap.cc:654] T 5fa6a43ca766412fbd638c20ea041949 P 73148a30e9e54a29b9ab2894b5f3f8ae: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:45.066267 8793 tablet_bootstrap.cc:654] T 5fa6a43ca766412fbd638c20ea041949 P 0f20308d9e324531a57e487bd730ba8f: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:45.067260 8794 tablet_bootstrap.cc:492] T 5fa6a43ca766412fbd638c20ea041949 P 2c35196c6d7d417b936d34cd9d912d4e: No bootstrap required, opened a new log
I20260621 07:47:45.067340 8794 ts_tablet_manager.cc:1397] T 5fa6a43ca766412fbd638c20ea041949 P 2c35196c6d7d417b936d34cd9d912d4e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 07:47:45.067489 8793 tablet_bootstrap.cc:492] T 5fa6a43ca766412fbd638c20ea041949 P 0f20308d9e324531a57e487bd730ba8f: No bootstrap required, opened a new log
I20260621 07:47:45.067562 8793 ts_tablet_manager.cc:1397] T 5fa6a43ca766412fbd638c20ea041949 P 0f20308d9e324531a57e487bd730ba8f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260621 07:47:45.067756 8794 raft_consensus.cc:348] T 5fa6a43ca766412fbd638c20ea041949 P 2c35196c6d7d417b936d34cd9d912d4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } }
I20260621 07:47:45.067855 8794 raft_consensus.cc:374] T 5fa6a43ca766412fbd638c20ea041949 P 2c35196c6d7d417b936d34cd9d912d4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:45.067891 8794 raft_consensus.cc:729] T 5fa6a43ca766412fbd638c20ea041949 P 2c35196c6d7d417b936d34cd9d912d4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c35196c6d7d417b936d34cd9d912d4e, State: Initialized, Role: FOLLOWER
I20260621 07:47:45.067925 8793 raft_consensus.cc:348] T 5fa6a43ca766412fbd638c20ea041949 P 0f20308d9e324531a57e487bd730ba8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } }
I20260621 07:47:45.068027 8793 raft_consensus.cc:374] T 5fa6a43ca766412fbd638c20ea041949 P 0f20308d9e324531a57e487bd730ba8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:45.068008 8794 consensus_queue.cc:260] T 5fa6a43ca766412fbd638c20ea041949 P 2c35196c6d7d417b936d34cd9d912d4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } }
I20260621 07:47:45.068073 8793 raft_consensus.cc:729] T 5fa6a43ca766412fbd638c20ea041949 P 0f20308d9e324531a57e487bd730ba8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f20308d9e324531a57e487bd730ba8f, State: Initialized, Role: FOLLOWER
I20260621 07:47:45.068193 8793 consensus_queue.cc:260] T 5fa6a43ca766412fbd638c20ea041949 P 0f20308d9e324531a57e487bd730ba8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } }
I20260621 07:47:45.068311 8794 ts_tablet_manager.cc:1428] T 5fa6a43ca766412fbd638c20ea041949 P 2c35196c6d7d417b936d34cd9d912d4e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260621 07:47:45.068463 8793 ts_tablet_manager.cc:1428] T 5fa6a43ca766412fbd638c20ea041949 P 0f20308d9e324531a57e487bd730ba8f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260621 07:47:45.068538 8800 tablet_replica.cc:439] T 5fa6a43ca766412fbd638c20ea041949 P 2c35196c6d7d417b936d34cd9d912d4e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } } }
I20260621 07:47:45.068701 8800 tablet_replica.cc:442] T 5fa6a43ca766412fbd638c20ea041949 P 2c35196c6d7d417b936d34cd9d912d4e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:47:45.068656 8799 tablet_replica.cc:439] T 5fa6a43ca766412fbd638c20ea041949 P 0f20308d9e324531a57e487bd730ba8f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } } }
I20260621 07:47:45.068753 8799 tablet_replica.cc:442] T 5fa6a43ca766412fbd638c20ea041949 P 0f20308d9e324531a57e487bd730ba8f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:47:45.073148 8795 tablet_bootstrap.cc:492] T 5fa6a43ca766412fbd638c20ea041949 P 73148a30e9e54a29b9ab2894b5f3f8ae: No bootstrap required, opened a new log
I20260621 07:47:45.073220 8795 ts_tablet_manager.cc:1397] T 5fa6a43ca766412fbd638c20ea041949 P 73148a30e9e54a29b9ab2894b5f3f8ae: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260621 07:47:45.073568 8795 raft_consensus.cc:348] T 5fa6a43ca766412fbd638c20ea041949 P 73148a30e9e54a29b9ab2894b5f3f8ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } }
I20260621 07:47:45.073660 8795 raft_consensus.cc:374] T 5fa6a43ca766412fbd638c20ea041949 P 73148a30e9e54a29b9ab2894b5f3f8ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:45.073683 8795 raft_consensus.cc:729] T 5fa6a43ca766412fbd638c20ea041949 P 73148a30e9e54a29b9ab2894b5f3f8ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73148a30e9e54a29b9ab2894b5f3f8ae, State: Initialized, Role: FOLLOWER
I20260621 07:47:45.073776 8795 consensus_queue.cc:260] T 5fa6a43ca766412fbd638c20ea041949 P 73148a30e9e54a29b9ab2894b5f3f8ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } }
I20260621 07:47:45.073985 8795 ts_tablet_manager.cc:1428] T 5fa6a43ca766412fbd638c20ea041949 P 73148a30e9e54a29b9ab2894b5f3f8ae: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260621 07:47:45.073997 8801 tablet_replica.cc:439] T 5fa6a43ca766412fbd638c20ea041949 P 73148a30e9e54a29b9ab2894b5f3f8ae: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } } }
I20260621 07:47:45.074148 8801 tablet_replica.cc:442] T 5fa6a43ca766412fbd638c20ea041949 P 73148a30e9e54a29b9ab2894b5f3f8ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:47:45.149300 8801 raft_consensus.cc:482] T 5fa6a43ca766412fbd638c20ea041949 P 73148a30e9e54a29b9ab2894b5f3f8ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:47:45.149436 8801 raft_consensus.cc:504] T 5fa6a43ca766412fbd638c20ea041949 P 73148a30e9e54a29b9ab2894b5f3f8ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } }
I20260621 07:47:45.150501 8801 leader_election.cc:288] T 5fa6a43ca766412fbd638c20ea041949 P 73148a30e9e54a29b9ab2894b5f3f8ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f20308d9e324531a57e487bd730ba8f (127.0.192.194:43285), 2c35196c6d7d417b936d34cd9d912d4e (127.0.192.195:44737)
I20260621 07:47:45.153888 8790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:45.150830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:43047 (local address 127.0.192.195:44737)
0621 07:47:45.151002 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:45.151007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:45.151059 (+ 52us) server_negotiation.cc:407] Connection header received
0621 07:47:45.151213 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:45.151216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:45.151260 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:45.151335 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:45.151587 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:45.152039 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:45.153058 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:45.153396 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:45.153512 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:45.153563 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:45.153733 (+ 170us) server_negotiation.cc:299] Negotiation successful
0621 07:47:45.153791 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260621 07:47:45.153920 8832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:45.150708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.195:44737 (local address 127.0.192.193:43047)
0621 07:47:45.150987 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:45.151000 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:45.151101 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:45.151347 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:45.151349 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:45.151363 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:45.151475 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:45.151485 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:45.152141 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:45.152145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:45.152938 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:45.152946 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:45.153502 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:45.153506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:45.153605 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:45.153624 (+ 19us) client_negotiation.cc:769] Sending connection context
0621 07:47:45.153709 (+ 85us) client_negotiation.cc:240] Negotiation successful
0621 07:47:45.153769 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":1,"thread_start_us":103,"threads_started":1}
I20260621 07:47:45.154127 8788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:45.150792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:53699 (local address 127.0.192.194:43285)
0621 07:47:45.150914 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:45.150919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:45.151166 (+ 247us) server_negotiation.cc:407] Connection header received
0621 07:47:45.151311 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:45.151316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:45.151363 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:45.151454 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:45.151712 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:45.152157 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:45.153181 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:45.153552 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:45.153680 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:45.153730 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:45.153914 (+ 184us) server_negotiation.cc:299] Negotiation successful
0621 07:47:45.153985 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260621 07:47:45.154313 8833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:45.150713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:43285 (local address 127.0.192.193:53699)
0621 07:47:45.151067 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:45.151078 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:45.151196 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:45.151457 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:45.151460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:45.151476 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:45.151602 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:45.151608 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:45.152269 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:45.152275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:45.153054 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:45.153063 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:45.153671 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:45.153675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:45.153770 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:45.153798 (+ 28us) client_negotiation.cc:769] Sending connection context
0621 07:47:45.153883 (+ 85us) client_negotiation.cc:240] Negotiation successful
0621 07:47:45.154088 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":36,"thread_start_us":121,"threads_started":1}
I20260621 07:47:45.154294 8719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fa6a43ca766412fbd638c20ea041949" candidate_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c35196c6d7d417b936d34cd9d912d4e" is_pre_election: true
I20260621 07:47:45.154429 8719 raft_consensus.cc:2454] T 5fa6a43ca766412fbd638c20ea041949 P 2c35196c6d7d417b936d34cd9d912d4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73148a30e9e54a29b9ab2894b5f3f8ae in term 0.
I20260621 07:47:45.154740 8371 leader_election.cc:302] T 5fa6a43ca766412fbd638c20ea041949 P 73148a30e9e54a29b9ab2894b5f3f8ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c35196c6d7d417b936d34cd9d912d4e, 73148a30e9e54a29b9ab2894b5f3f8ae; no voters:
I20260621 07:47:45.154896 8579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fa6a43ca766412fbd638c20ea041949" candidate_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f20308d9e324531a57e487bd730ba8f" is_pre_election: true
I20260621 07:47:45.154973 8801 raft_consensus.cc:2781] T 5fa6a43ca766412fbd638c20ea041949 P 73148a30e9e54a29b9ab2894b5f3f8ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:47:45.155030 8801 raft_consensus.cc:482] T 5fa6a43ca766412fbd638c20ea041949 P 73148a30e9e54a29b9ab2894b5f3f8ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:47:45.155069 8801 raft_consensus.cc:3037] T 5fa6a43ca766412fbd638c20ea041949 P 73148a30e9e54a29b9ab2894b5f3f8ae [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:45.155134 8579 raft_consensus.cc:2454] T 5fa6a43ca766412fbd638c20ea041949 P 0f20308d9e324531a57e487bd730ba8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73148a30e9e54a29b9ab2894b5f3f8ae in term 0.
I20260621 07:47:45.156256 8801 raft_consensus.cc:504] T 5fa6a43ca766412fbd638c20ea041949 P 73148a30e9e54a29b9ab2894b5f3f8ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } }
I20260621 07:47:45.156646 8801 leader_election.cc:288] T 5fa6a43ca766412fbd638c20ea041949 P 73148a30e9e54a29b9ab2894b5f3f8ae [CANDIDATE]: Term 1 election: Requested vote from peers 0f20308d9e324531a57e487bd730ba8f (127.0.192.194:43285), 2c35196c6d7d417b936d34cd9d912d4e (127.0.192.195:44737)
I20260621 07:47:45.156883 8579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fa6a43ca766412fbd638c20ea041949" candidate_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f20308d9e324531a57e487bd730ba8f"
I20260621 07:47:45.156960 8719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fa6a43ca766412fbd638c20ea041949" candidate_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c35196c6d7d417b936d34cd9d912d4e"
I20260621 07:47:45.157001 8579 raft_consensus.cc:3037] T 5fa6a43ca766412fbd638c20ea041949 P 0f20308d9e324531a57e487bd730ba8f [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:45.157064 8719 raft_consensus.cc:3037] T 5fa6a43ca766412fbd638c20ea041949 P 2c35196c6d7d417b936d34cd9d912d4e [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:45.158037 8579 raft_consensus.cc:2454] T 5fa6a43ca766412fbd638c20ea041949 P 0f20308d9e324531a57e487bd730ba8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73148a30e9e54a29b9ab2894b5f3f8ae in term 1.
I20260621 07:47:45.158030 8719 raft_consensus.cc:2454] T 5fa6a43ca766412fbd638c20ea041949 P 2c35196c6d7d417b936d34cd9d912d4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73148a30e9e54a29b9ab2894b5f3f8ae in term 1.
I20260621 07:47:45.158279 8371 leader_election.cc:302] T 5fa6a43ca766412fbd638c20ea041949 P 73148a30e9e54a29b9ab2894b5f3f8ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c35196c6d7d417b936d34cd9d912d4e, 73148a30e9e54a29b9ab2894b5f3f8ae; no voters:
I20260621 07:47:45.158550 8801 raft_consensus.cc:2781] T 5fa6a43ca766412fbd638c20ea041949 P 73148a30e9e54a29b9ab2894b5f3f8ae [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:47:45.158716 8801 raft_consensus.cc:686] T 5fa6a43ca766412fbd638c20ea041949 P 73148a30e9e54a29b9ab2894b5f3f8ae [term 1 LEADER]: Becoming Leader. State: Replica: 73148a30e9e54a29b9ab2894b5f3f8ae, State: Running, Role: LEADER
I20260621 07:47:45.158951 8801 consensus_queue.cc:237] T 5fa6a43ca766412fbd638c20ea041949 P 73148a30e9e54a29b9ab2894b5f3f8ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } }
I20260621 07:47:45.160221 8834 tablet_replica.cc:439] T 5fa6a43ca766412fbd638c20ea041949 P 73148a30e9e54a29b9ab2894b5f3f8ae: TxnStatusTablet state changed. Reason: New leader 73148a30e9e54a29b9ab2894b5f3f8ae. Latest consensus state: current_term: 1 leader_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } } }
I20260621 07:47:45.160324 8834 tablet_replica.cc:442] T 5fa6a43ca766412fbd638c20ea041949 P 73148a30e9e54a29b9ab2894b5f3f8ae: This TxnStatusTablet replica's current role is: LEADER
I20260621 07:47:45.160549 8836 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260621 07:47:45.160722 8836 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260621 07:47:45.161360 8281 catalog_manager.cc:5495] T 5fa6a43ca766412fbd638c20ea041949 P 73148a30e9e54a29b9ab2894b5f3f8ae reported cstate change: term changed from 0 to 1, leader changed from <none> to 73148a30e9e54a29b9ab2894b5f3f8ae (127.0.192.193). New cstate: current_term: 1 leader_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } health_report { overall_health: UNKNOWN } } }
I20260621 07:47:45.172365 8792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:45.168838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45648 (local address 127.0.192.193:33397)
0621 07:47:45.168939 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:45.168943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:45.169090 (+ 147us) server_negotiation.cc:407] Connection header received
0621 07:47:45.169266 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:45.169268 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:45.169311 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:45.169387 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:47:45.169672 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:45.170100 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:45.171186 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:45.171359 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:45.171423 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:45.171731 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:47:45.172055 (+ 324us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:47:45.172129 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:45.172255 (+ 126us) server_negotiation.cc:299] Negotiation successful
0621 07:47:45.172289 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260621 07:47:45.172461 8828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:45.168774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:33397 (local address 127.0.0.1:45648)
0621 07:47:45.169011 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:45.169028 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:45.169144 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:45.169433 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:45.169436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:45.169463 (+ 27us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 07:47:45.169570 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:45.169577 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:45.170207 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:45.170210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:45.171076 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:45.171087 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:45.171463 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:45.171467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:45.171561 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:45.171609 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 07:47:45.172147 (+ 538us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 07:47:45.172161 (+ 14us) client_negotiation.cc:769] Sending connection context
0621 07:47:45.172241 (+ 80us) client_negotiation.cc:240] Negotiation successful
0621 07:47:45.172317 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260621 07:47:45.174790 8719 raft_consensus.cc:1261] T 5fa6a43ca766412fbd638c20ea041949 P 2c35196c6d7d417b936d34cd9d912d4e [term 1 FOLLOWER]: Refusing update from remote peer 73148a30e9e54a29b9ab2894b5f3f8ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:47:45.174918 8579 raft_consensus.cc:1261] T 5fa6a43ca766412fbd638c20ea041949 P 0f20308d9e324531a57e487bd730ba8f [term 1 FOLLOWER]: Refusing update from remote peer 73148a30e9e54a29b9ab2894b5f3f8ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:47:45.175251 8801 consensus_queue.cc:1035] T 5fa6a43ca766412fbd638c20ea041949 P 73148a30e9e54a29b9ab2894b5f3f8ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:47:45.175396 8834 consensus_queue.cc:1035] T 5fa6a43ca766412fbd638c20ea041949 P 73148a30e9e54a29b9ab2894b5f3f8ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:47:45.176910 8799 tablet_replica.cc:439] T 5fa6a43ca766412fbd638c20ea041949 P 0f20308d9e324531a57e487bd730ba8f: TxnStatusTablet state changed. Reason: New leader 73148a30e9e54a29b9ab2894b5f3f8ae. Latest consensus state: current_term: 1 leader_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } } }
I20260621 07:47:45.177009 8799 tablet_replica.cc:442] T 5fa6a43ca766412fbd638c20ea041949 P 0f20308d9e324531a57e487bd730ba8f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:47:45.177376 8800 tablet_replica.cc:439] T 5fa6a43ca766412fbd638c20ea041949 P 2c35196c6d7d417b936d34cd9d912d4e: TxnStatusTablet state changed. Reason: New leader 73148a30e9e54a29b9ab2894b5f3f8ae. Latest consensus state: current_term: 1 leader_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } } }
I20260621 07:47:45.177512 8800 tablet_replica.cc:442] T 5fa6a43ca766412fbd638c20ea041949 P 2c35196c6d7d417b936d34cd9d912d4e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:47:45.178683 8834 tablet_replica.cc:439] T 5fa6a43ca766412fbd638c20ea041949 P 73148a30e9e54a29b9ab2894b5f3f8ae: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } } }
I20260621 07:47:45.178778 8834 tablet_replica.cc:442] T 5fa6a43ca766412fbd638c20ea041949 P 73148a30e9e54a29b9ab2894b5f3f8ae: This TxnStatusTablet replica's current role is: LEADER
I20260621 07:47:45.179538 8799 tablet_replica.cc:439] T 5fa6a43ca766412fbd638c20ea041949 P 0f20308d9e324531a57e487bd730ba8f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } } }
I20260621 07:47:45.179641 8799 tablet_replica.cc:442] T 5fa6a43ca766412fbd638c20ea041949 P 0f20308d9e324531a57e487bd730ba8f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:47:45.183362 8834 tablet_replica.cc:439] T 5fa6a43ca766412fbd638c20ea041949 P 73148a30e9e54a29b9ab2894b5f3f8ae: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } } }
I20260621 07:47:45.183444 8834 tablet_replica.cc:442] T 5fa6a43ca766412fbd638c20ea041949 P 73148a30e9e54a29b9ab2894b5f3f8ae: This TxnStatusTablet replica's current role is: LEADER
I20260621 07:47:45.184389 8800 tablet_replica.cc:439] T 5fa6a43ca766412fbd638c20ea041949 P 2c35196c6d7d417b936d34cd9d912d4e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } } }
I20260621 07:47:45.184494 8800 tablet_replica.cc:442] T 5fa6a43ca766412fbd638c20ea041949 P 2c35196c6d7d417b936d34cd9d912d4e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:47:45.191234 8792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:45.188085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:54689 (local address 127.0.192.193:33397)
0621 07:47:45.188164 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:45.188169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:45.188607 (+ 438us) server_negotiation.cc:407] Connection header received
0621 07:47:45.188653 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:45.188659 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:45.188707 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:45.188776 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:47:45.189011 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:45.189586 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:45.190344 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:45.190551 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:45.190649 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:45.190883 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:47:45.190951 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:47:45.191019 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:45.191099 (+ 80us) server_negotiation.cc:299] Negotiation successful
0621 07:47:45.191136 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260621 07:47:45.191282 8844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:45.188034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:33397 (local address 127.0.192.194:54689)
0621 07:47:45.188282 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:45.188294 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:45.188433 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:45.188804 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:45.188807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:45.188815 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 07:47:45.188913 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:45.188918 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:45.189678 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:45.189684 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:45.190236 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:45.190244 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:45.190649 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:45.190653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:45.190728 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:45.190769 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 07:47:45.191032 (+ 263us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 07:47:45.191038 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 07:47:45.191092 (+ 54us) client_negotiation.cc:240] Negotiation successful
0621 07:47:45.191163 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":92,"threads_started":1}
I20260621 07:47:45.197429 8439 raft_consensus.cc:1261] T c2182cf9de6e469ead99160fcc31dabe P 73148a30e9e54a29b9ab2894b5f3f8ae [term 1 FOLLOWER]: Refusing update from remote peer 0f20308d9e324531a57e487bd730ba8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:47:45.197427 8719 raft_consensus.cc:1261] T c2182cf9de6e469ead99160fcc31dabe P 2c35196c6d7d417b936d34cd9d912d4e [term 1 FOLLOWER]: Refusing update from remote peer 0f20308d9e324531a57e487bd730ba8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:47:45.197798 8813 consensus_queue.cc:1035] T c2182cf9de6e469ead99160fcc31dabe P 0f20308d9e324531a57e487bd730ba8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:47:45.198029 8813 consensus_queue.cc:1035] T c2182cf9de6e469ead99160fcc31dabe P 0f20308d9e324531a57e487bd730ba8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:47:45.219110 8848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:45.211745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:43285 (local address 127.0.192.193:56113)
0621 07:47:45.212005 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:45.212023 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:45.212170 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:45.212380 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:45.212382 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:45.212399 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 07:47:45.212496 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:45.212516 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:45.213490 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:45.213494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:45.217902 (+ 4408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:45.217912 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:45.218475 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:45.218478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:45.218541 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:45.218587 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 07:47:45.218876 (+ 289us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 07:47:45.218881 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 07:47:45.218928 (+ 47us) client_negotiation.cc:240] Negotiation successful
0621 07:47:45.218982 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":106,"threads_started":1}
I20260621 07:47:45.219125 8788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:45.211807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:56113 (local address 127.0.192.194:43285)
0621 07:47:45.211909 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:45.211913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:45.212106 (+ 193us) server_negotiation.cc:407] Connection header received
0621 07:47:45.212248 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:45.212252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:45.212289 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:45.212370 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:47:45.212678 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:45.213342 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:45.218009 (+ 4667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:45.218358 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:45.218480 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:45.218710 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:47:45.218789 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:47:45.218870 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:45.218941 (+ 71us) server_negotiation.cc:299] Negotiation successful
0621 07:47:45.218993 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260621 07:47:45.239679 8282 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:47:45.239840 8282 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:47:45.241683 8282 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a5a7e6a80b74aebbeb361add74240ab: Sending DeleteTablet for 3 replicas of tablet c2182cf9de6e469ead99160fcc31dabe
I20260621 07:47:45.242106 8413 tablet_service.cc:1514] Processing DeleteTablet for tablet c2182cf9de6e469ead99160fcc31dabe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:47:45 UTC) from {username='oryx', principal='oryx/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:45632
I20260621 07:47:45.242182 8559 tablet_service.cc:1514] Processing DeleteTablet for tablet c2182cf9de6e469ead99160fcc31dabe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:47:45 UTC) from {username='oryx', principal='oryx/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:53012
I20260621 07:47:45.242334 8816 tablet_replica.cc:330] T c2182cf9de6e469ead99160fcc31dabe P 73148a30e9e54a29b9ab2894b5f3f8ae: stopping tablet replica
I20260621 07:47:45.242331 8818 tablet_replica.cc:330] T c2182cf9de6e469ead99160fcc31dabe P 0f20308d9e324531a57e487bd730ba8f: stopping tablet replica
I20260621 07:47:45.242329 8699 tablet_service.cc:1514] Processing DeleteTablet for tablet c2182cf9de6e469ead99160fcc31dabe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:47:45 UTC) from {username='oryx', principal='oryx/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:58036
I20260621 07:47:45.242429 8818 raft_consensus.cc:2229] T c2182cf9de6e469ead99160fcc31dabe P 0f20308d9e324531a57e487bd730ba8f [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:47:45.242424 8816 raft_consensus.cc:2229] T c2182cf9de6e469ead99160fcc31dabe P 73148a30e9e54a29b9ab2894b5f3f8ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:47:45.242496 8817 tablet_replica.cc:330] T c2182cf9de6e469ead99160fcc31dabe P 2c35196c6d7d417b936d34cd9d912d4e: stopping tablet replica
I20260621 07:47:45.242507 8816 raft_consensus.cc:2258] T c2182cf9de6e469ead99160fcc31dabe P 73148a30e9e54a29b9ab2894b5f3f8ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:47:45.242581 8818 raft_consensus.cc:2258] T c2182cf9de6e469ead99160fcc31dabe P 0f20308d9e324531a57e487bd730ba8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:47:45.242592 8817 raft_consensus.cc:2229] T c2182cf9de6e469ead99160fcc31dabe P 2c35196c6d7d417b936d34cd9d912d4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:47:45.242683 8817 raft_consensus.cc:2258] T c2182cf9de6e469ead99160fcc31dabe P 2c35196c6d7d417b936d34cd9d912d4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:47:45.243155 8816 ts_tablet_manager.cc:1905] T c2182cf9de6e469ead99160fcc31dabe P 73148a30e9e54a29b9ab2894b5f3f8ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:47:45.243162 8818 ts_tablet_manager.cc:1905] T c2182cf9de6e469ead99160fcc31dabe P 0f20308d9e324531a57e487bd730ba8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:47:45.243429 8817 ts_tablet_manager.cc:1905] T c2182cf9de6e469ead99160fcc31dabe P 2c35196c6d7d417b936d34cd9d912d4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:47:45.245810 8818 ts_tablet_manager.cc:1918] T c2182cf9de6e469ead99160fcc31dabe P 0f20308d9e324531a57e487bd730ba8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 07:47:45.245880 8818 log.cc:1196] T c2182cf9de6e469ead99160fcc31dabe P 0f20308d9e324531a57e487bd730ba8f: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-1/wal/wals/c2182cf9de6e469ead99160fcc31dabe
I20260621 07:47:45.246075 8816 ts_tablet_manager.cc:1918] T c2182cf9de6e469ead99160fcc31dabe P 73148a30e9e54a29b9ab2894b5f3f8ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 07:47:45.246116 8818 ts_tablet_manager.cc:1939] T c2182cf9de6e469ead99160fcc31dabe P 0f20308d9e324531a57e487bd730ba8f: Deleting consensus metadata
I20260621 07:47:45.246133 8816 log.cc:1196] T c2182cf9de6e469ead99160fcc31dabe P 73148a30e9e54a29b9ab2894b5f3f8ae: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-0/wal/wals/c2182cf9de6e469ead99160fcc31dabe
I20260621 07:47:45.246206 8817 ts_tablet_manager.cc:1918] T c2182cf9de6e469ead99160fcc31dabe P 2c35196c6d7d417b936d34cd9d912d4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 07:47:45.246268 8817 log.cc:1196] T c2182cf9de6e469ead99160fcc31dabe P 2c35196c6d7d417b936d34cd9d912d4e: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-2/wal/wals/c2182cf9de6e469ead99160fcc31dabe
I20260621 07:47:45.246351 8816 ts_tablet_manager.cc:1939] T c2182cf9de6e469ead99160fcc31dabe P 73148a30e9e54a29b9ab2894b5f3f8ae: Deleting consensus metadata
I20260621 07:47:45.246501 8817 ts_tablet_manager.cc:1939] T c2182cf9de6e469ead99160fcc31dabe P 2c35196c6d7d417b936d34cd9d912d4e: Deleting consensus metadata
I20260621 07:47:45.246917 8266 catalog_manager.cc:4841] TS 73148a30e9e54a29b9ab2894b5f3f8ae (127.0.192.193:33397): tablet c2182cf9de6e469ead99160fcc31dabe (table test-table [id=30505adcfd874a01896b894f436662bd]) successfully deleted
I20260621 07:47:45.247064 8266 catalog_manager.cc:4841] TS 0f20308d9e324531a57e487bd730ba8f (127.0.192.194:43285): tablet c2182cf9de6e469ead99160fcc31dabe (table test-table [id=30505adcfd874a01896b894f436662bd]) successfully deleted
I20260621 07:47:45.247450 8267 catalog_manager.cc:4841] TS 2c35196c6d7d417b936d34cd9d912d4e (127.0.192.195:44737): tablet c2182cf9de6e469ead99160fcc31dabe (table test-table [id=30505adcfd874a01896b894f436662bd]) successfully deleted
Jun 21 07:47:45 dist-test-slave-lzx3 krb5kdc[8237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 07:47:45 dist-test-slave-lzx3 krb5kdc[8237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:47:45 dist-test-slave-lzx3 krb5kdc[8237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
W20260621 07:47:45.269045 8331 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:45.262744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46486 (local address 127.0.192.254:45399)
0621 07:47:45.262856 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:45.262860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:45.263002 (+ 142us) server_negotiation.cc:407] Connection header received
0621 07:47:45.263142 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:45.263146 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:45.263187 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:45.263265 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:45.263972 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:45.264481 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:45.265463 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:45.265700 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:45.265794 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:45.268910 (+ 3116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46486: BlockingRecv error: recv got EOF from 127.0.0.1:46486 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 21 07:47:45 dist-test-slave-lzx3 krb5kdc[8237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM
I20260621 07:47:45.277647 8331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:45.270272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46498 (local address 127.0.192.254:45399)
0621 07:47:45.270437 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:45.270441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:45.270541 (+ 100us) server_negotiation.cc:407] Connection header received
0621 07:47:45.270672 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:45.270675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:45.270714 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:45.270788 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:45.271286 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:45.271715 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:45.272393 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:45.272669 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:45.272763 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:45.274830 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:45.274833 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:45.274836 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:45.274857 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:45.276141 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:45.276540 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:45.276543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:45.276545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:45.276642 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:45.276912 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:45.276917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:45.276918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:45.277078 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:45.277201 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:45.277410 (+ 209us) server_negotiation.cc:299] Negotiation successful
0621 07:47:45.277509 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260621 07:47:45.280045 8282 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46498:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:47:45.280488 8282 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:47:45.284986 8413 tablet_service.cc:1467] Processing CreateTablet for tablet 730c29b74a4d40c4a34c32f18ac25952 (DEFAULT_TABLE table=test-table [id=55674ce587e84b10a7c37be75b542bbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:47:45.285077 8699 tablet_service.cc:1467] Processing CreateTablet for tablet 730c29b74a4d40c4a34c32f18ac25952 (DEFAULT_TABLE table=test-table [id=55674ce587e84b10a7c37be75b542bbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:47:45.285082 8559 tablet_service.cc:1467] Processing CreateTablet for tablet 730c29b74a4d40c4a34c32f18ac25952 (DEFAULT_TABLE table=test-table [id=55674ce587e84b10a7c37be75b542bbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:47:45.285252 8413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 730c29b74a4d40c4a34c32f18ac25952. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:45.285307 8699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 730c29b74a4d40c4a34c32f18ac25952. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:45.285307 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 730c29b74a4d40c4a34c32f18ac25952. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:45.287464 8794 tablet_bootstrap.cc:492] T 730c29b74a4d40c4a34c32f18ac25952 P 2c35196c6d7d417b936d34cd9d912d4e: Bootstrap starting.
I20260621 07:47:45.287501 8793 tablet_bootstrap.cc:492] T 730c29b74a4d40c4a34c32f18ac25952 P 0f20308d9e324531a57e487bd730ba8f: Bootstrap starting.
I20260621 07:47:45.287563 8795 tablet_bootstrap.cc:492] T 730c29b74a4d40c4a34c32f18ac25952 P 73148a30e9e54a29b9ab2894b5f3f8ae: Bootstrap starting.
I20260621 07:47:45.288200 8793 tablet_bootstrap.cc:654] T 730c29b74a4d40c4a34c32f18ac25952 P 0f20308d9e324531a57e487bd730ba8f: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:45.288218 8794 tablet_bootstrap.cc:654] T 730c29b74a4d40c4a34c32f18ac25952 P 2c35196c6d7d417b936d34cd9d912d4e: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:45.288419 8795 tablet_bootstrap.cc:654] T 730c29b74a4d40c4a34c32f18ac25952 P 73148a30e9e54a29b9ab2894b5f3f8ae: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:45.289516 8793 tablet_bootstrap.cc:492] T 730c29b74a4d40c4a34c32f18ac25952 P 0f20308d9e324531a57e487bd730ba8f: No bootstrap required, opened a new log
I20260621 07:47:45.289610 8793 ts_tablet_manager.cc:1397] T 730c29b74a4d40c4a34c32f18ac25952 P 0f20308d9e324531a57e487bd730ba8f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 07:47:45.289762 8794 tablet_bootstrap.cc:492] T 730c29b74a4d40c4a34c32f18ac25952 P 2c35196c6d7d417b936d34cd9d912d4e: No bootstrap required, opened a new log
I20260621 07:47:45.289821 8794 ts_tablet_manager.cc:1397] T 730c29b74a4d40c4a34c32f18ac25952 P 2c35196c6d7d417b936d34cd9d912d4e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 07:47:45.290014 8793 raft_consensus.cc:348] T 730c29b74a4d40c4a34c32f18ac25952 P 0f20308d9e324531a57e487bd730ba8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } }
I20260621 07:47:45.290144 8793 raft_consensus.cc:374] T 730c29b74a4d40c4a34c32f18ac25952 P 0f20308d9e324531a57e487bd730ba8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:45.290179 8793 raft_consensus.cc:729] T 730c29b74a4d40c4a34c32f18ac25952 P 0f20308d9e324531a57e487bd730ba8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f20308d9e324531a57e487bd730ba8f, State: Initialized, Role: FOLLOWER
I20260621 07:47:45.290290 8793 consensus_queue.cc:260] T 730c29b74a4d40c4a34c32f18ac25952 P 0f20308d9e324531a57e487bd730ba8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } }
I20260621 07:47:45.290524 8793 ts_tablet_manager.cc:1428] T 730c29b74a4d40c4a34c32f18ac25952 P 0f20308d9e324531a57e487bd730ba8f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260621 07:47:45.290496 8794 raft_consensus.cc:348] T 730c29b74a4d40c4a34c32f18ac25952 P 2c35196c6d7d417b936d34cd9d912d4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } }
I20260621 07:47:45.290601 8794 raft_consensus.cc:374] T 730c29b74a4d40c4a34c32f18ac25952 P 2c35196c6d7d417b936d34cd9d912d4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:45.290633 8794 raft_consensus.cc:729] T 730c29b74a4d40c4a34c32f18ac25952 P 2c35196c6d7d417b936d34cd9d912d4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c35196c6d7d417b936d34cd9d912d4e, State: Initialized, Role: FOLLOWER
I20260621 07:47:45.290758 8794 consensus_queue.cc:260] T 730c29b74a4d40c4a34c32f18ac25952 P 2c35196c6d7d417b936d34cd9d912d4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } }
I20260621 07:47:45.290988 8794 ts_tablet_manager.cc:1428] T 730c29b74a4d40c4a34c32f18ac25952 P 2c35196c6d7d417b936d34cd9d912d4e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260621 07:47:45.291352 8795 tablet_bootstrap.cc:492] T 730c29b74a4d40c4a34c32f18ac25952 P 73148a30e9e54a29b9ab2894b5f3f8ae: No bootstrap required, opened a new log
I20260621 07:47:45.291430 8795 ts_tablet_manager.cc:1397] T 730c29b74a4d40c4a34c32f18ac25952 P 73148a30e9e54a29b9ab2894b5f3f8ae: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260621 07:47:45.291829 8795 raft_consensus.cc:348] T 730c29b74a4d40c4a34c32f18ac25952 P 73148a30e9e54a29b9ab2894b5f3f8ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } }
I20260621 07:47:45.291968 8795 raft_consensus.cc:374] T 730c29b74a4d40c4a34c32f18ac25952 P 73148a30e9e54a29b9ab2894b5f3f8ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:45.292006 8795 raft_consensus.cc:729] T 730c29b74a4d40c4a34c32f18ac25952 P 73148a30e9e54a29b9ab2894b5f3f8ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73148a30e9e54a29b9ab2894b5f3f8ae, State: Initialized, Role: FOLLOWER
I20260621 07:47:45.292114 8795 consensus_queue.cc:260] T 730c29b74a4d40c4a34c32f18ac25952 P 73148a30e9e54a29b9ab2894b5f3f8ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } }
I20260621 07:47:45.292356 8795 ts_tablet_manager.cc:1428] T 730c29b74a4d40c4a34c32f18ac25952 P 73148a30e9e54a29b9ab2894b5f3f8ae: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260621 07:47:45.492762 8800 raft_consensus.cc:482] T 730c29b74a4d40c4a34c32f18ac25952 P 2c35196c6d7d417b936d34cd9d912d4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:47:45.492909 8800 raft_consensus.cc:504] T 730c29b74a4d40c4a34c32f18ac25952 P 2c35196c6d7d417b936d34cd9d912d4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } }
I20260621 07:47:45.493978 8800 leader_election.cc:288] T 730c29b74a4d40c4a34c32f18ac25952 P 2c35196c6d7d417b936d34cd9d912d4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73148a30e9e54a29b9ab2894b5f3f8ae (127.0.192.193:33397), 0f20308d9e324531a57e487bd730ba8f (127.0.192.194:43285)
I20260621 07:47:45.497746 8792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:45.494296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:40655 (local address 127.0.192.193:33397)
0621 07:47:45.494419 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:45.494423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:45.494557 (+ 134us) server_negotiation.cc:407] Connection header received
0621 07:47:45.494726 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:45.494729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:45.494792 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:45.494836 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:45.495183 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:45.495717 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:45.496825 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:45.497213 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:45.497349 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:45.497403 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:45.497573 (+ 170us) server_negotiation.cc:299] Negotiation successful
0621 07:47:45.497641 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260621 07:47:45.497766 8865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:45.494219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:33397 (local address 127.0.192.195:40655)
0621 07:47:45.494453 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:45.494469 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:45.494575 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:45.494959 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:45.494961 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:45.494976 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:45.495077 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:45.495081 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:45.495817 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:45.495819 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:45.496708 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:45.496716 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:45.497348 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:45.497352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:45.497442 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:45.497467 (+ 25us) client_negotiation.cc:769] Sending connection context
0621 07:47:45.497553 (+ 86us) client_negotiation.cc:240] Negotiation successful
0621 07:47:45.497621 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":90,"threads_started":1}
I20260621 07:47:45.498234 8439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "730c29b74a4d40c4a34c32f18ac25952" candidate_uuid: "2c35196c6d7d417b936d34cd9d912d4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" is_pre_election: true
I20260621 07:47:45.498401 8439 raft_consensus.cc:2454] T 730c29b74a4d40c4a34c32f18ac25952 P 73148a30e9e54a29b9ab2894b5f3f8ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c35196c6d7d417b936d34cd9d912d4e in term 0.
I20260621 07:47:45.498679 8866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:45.494584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:43285 (local address 127.0.192.195:48283)
0621 07:47:45.494793 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:45.494807 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:45.494911 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:45.495834 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:45.495836 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:45.495852 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:47:45.495947 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:45.495951 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:45.496740 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:45.496743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:45.497467 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:45.497475 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:45.498216 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:45.498219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:45.498311 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:45.498342 (+ 31us) client_negotiation.cc:769] Sending connection context
0621 07:47:45.498456 (+ 114us) client_negotiation.cc:240] Negotiation successful
0621 07:47:45.498517 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":74,"threads_started":1}
I20260621 07:47:45.498739 8788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:45.494689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:48283 (local address 127.0.192.194:43285)
0621 07:47:45.494801 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:45.494807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:45.494865 (+ 58us) server_negotiation.cc:407] Connection header received
0621 07:47:45.495024 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:45.495027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:45.495082 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:45.495169 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:47:45.496066 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:45.496614 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:45.497593 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:45.498118 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:45.498227 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:45.498271 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:45.498532 (+ 261us) server_negotiation.cc:299] Negotiation successful
0621 07:47:45.498619 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 07:47:45.498800 8650 leader_election.cc:302] T 730c29b74a4d40c4a34c32f18ac25952 P 2c35196c6d7d417b936d34cd9d912d4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c35196c6d7d417b936d34cd9d912d4e, 73148a30e9e54a29b9ab2894b5f3f8ae; no voters:
I20260621 07:47:45.499060 8800 raft_consensus.cc:2781] T 730c29b74a4d40c4a34c32f18ac25952 P 2c35196c6d7d417b936d34cd9d912d4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:47:45.499130 8800 raft_consensus.cc:482] T 730c29b74a4d40c4a34c32f18ac25952 P 2c35196c6d7d417b936d34cd9d912d4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:47:45.499177 8800 raft_consensus.cc:3037] T 730c29b74a4d40c4a34c32f18ac25952 P 2c35196c6d7d417b936d34cd9d912d4e [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:45.499367 8579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "730c29b74a4d40c4a34c32f18ac25952" candidate_uuid: "2c35196c6d7d417b936d34cd9d912d4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f20308d9e324531a57e487bd730ba8f" is_pre_election: true
I20260621 07:47:45.499531 8579 raft_consensus.cc:2454] T 730c29b74a4d40c4a34c32f18ac25952 P 0f20308d9e324531a57e487bd730ba8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c35196c6d7d417b936d34cd9d912d4e in term 0.
I20260621 07:47:45.500319 8800 raft_consensus.cc:504] T 730c29b74a4d40c4a34c32f18ac25952 P 2c35196c6d7d417b936d34cd9d912d4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } }
I20260621 07:47:45.500681 8800 leader_election.cc:288] T 730c29b74a4d40c4a34c32f18ac25952 P 2c35196c6d7d417b936d34cd9d912d4e [CANDIDATE]: Term 1 election: Requested vote from peers 73148a30e9e54a29b9ab2894b5f3f8ae (127.0.192.193:33397), 0f20308d9e324531a57e487bd730ba8f (127.0.192.194:43285)
I20260621 07:47:45.501003 8439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "730c29b74a4d40c4a34c32f18ac25952" candidate_uuid: "2c35196c6d7d417b936d34cd9d912d4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae"
I20260621 07:47:45.501030 8579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "730c29b74a4d40c4a34c32f18ac25952" candidate_uuid: "2c35196c6d7d417b936d34cd9d912d4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f20308d9e324531a57e487bd730ba8f"
I20260621 07:47:45.501109 8439 raft_consensus.cc:3037] T 730c29b74a4d40c4a34c32f18ac25952 P 73148a30e9e54a29b9ab2894b5f3f8ae [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:45.501197 8579 raft_consensus.cc:3037] T 730c29b74a4d40c4a34c32f18ac25952 P 0f20308d9e324531a57e487bd730ba8f [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:45.501955 8439 raft_consensus.cc:2454] T 730c29b74a4d40c4a34c32f18ac25952 P 73148a30e9e54a29b9ab2894b5f3f8ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c35196c6d7d417b936d34cd9d912d4e in term 1.
I20260621 07:47:45.502138 8579 raft_consensus.cc:2454] T 730c29b74a4d40c4a34c32f18ac25952 P 0f20308d9e324531a57e487bd730ba8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c35196c6d7d417b936d34cd9d912d4e in term 1.
I20260621 07:47:45.502218 8650 leader_election.cc:302] T 730c29b74a4d40c4a34c32f18ac25952 P 2c35196c6d7d417b936d34cd9d912d4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c35196c6d7d417b936d34cd9d912d4e, 73148a30e9e54a29b9ab2894b5f3f8ae; no voters:
I20260621 07:47:45.502373 8800 raft_consensus.cc:2781] T 730c29b74a4d40c4a34c32f18ac25952 P 2c35196c6d7d417b936d34cd9d912d4e [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:47:45.502605 8800 raft_consensus.cc:686] T 730c29b74a4d40c4a34c32f18ac25952 P 2c35196c6d7d417b936d34cd9d912d4e [term 1 LEADER]: Becoming Leader. State: Replica: 2c35196c6d7d417b936d34cd9d912d4e, State: Running, Role: LEADER
I20260621 07:47:45.502842 8800 consensus_queue.cc:237] T 730c29b74a4d40c4a34c32f18ac25952 P 2c35196c6d7d417b936d34cd9d912d4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } }
I20260621 07:47:45.505107 8280 catalog_manager.cc:5495] T 730c29b74a4d40c4a34c32f18ac25952 P 2c35196c6d7d417b936d34cd9d912d4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c35196c6d7d417b936d34cd9d912d4e (127.0.192.195). New cstate: current_term: 1 leader_uuid: "2c35196c6d7d417b936d34cd9d912d4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } health_report { overall_health: HEALTHY } } }
I20260621 07:47:45.527863 8790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:45.524231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58046 (local address 127.0.192.195:44737)
0621 07:47:45.524348 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:45.524352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:45.524366 (+ 14us) server_negotiation.cc:407] Connection header received
0621 07:47:45.524453 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:45.524456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:45.524511 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:45.524642 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:47:45.524839 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:45.525556 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:45.526364 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:45.526696 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:45.526791 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:45.527056 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:47:45.527531 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:47:45.527634 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:45.527683 (+ 49us) server_negotiation.cc:299] Negotiation successful
0621 07:47:45.527731 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260621 07:47:45.530272 8579 raft_consensus.cc:1261] T 730c29b74a4d40c4a34c32f18ac25952 P 0f20308d9e324531a57e487bd730ba8f [term 1 FOLLOWER]: Refusing update from remote peer 2c35196c6d7d417b936d34cd9d912d4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:47:45.530268 8439 raft_consensus.cc:1261] T 730c29b74a4d40c4a34c32f18ac25952 P 73148a30e9e54a29b9ab2894b5f3f8ae [term 1 FOLLOWER]: Refusing update from remote peer 2c35196c6d7d417b936d34cd9d912d4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:47:45.530735 8867 consensus_queue.cc:1035] T 730c29b74a4d40c4a34c32f18ac25952 P 2c35196c6d7d417b936d34cd9d912d4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:47:45.530949 8800 consensus_queue.cc:1035] T 730c29b74a4d40c4a34c32f18ac25952 P 2c35196c6d7d417b936d34cd9d912d4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:47:45.540107 8282 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:47:45.540207 8282 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:47:45.542152 8282 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a5a7e6a80b74aebbeb361add74240ab: Sending DeleteTablet for 3 replicas of tablet 730c29b74a4d40c4a34c32f18ac25952
I20260621 07:47:45.542768 8699 tablet_service.cc:1514] Processing DeleteTablet for tablet 730c29b74a4d40c4a34c32f18ac25952 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:47:45 UTC) from {username='oryx', principal='oryx/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:58036
I20260621 07:47:45.542768 8559 tablet_service.cc:1514] Processing DeleteTablet for tablet 730c29b74a4d40c4a34c32f18ac25952 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:47:45 UTC) from {username='oryx', principal='oryx/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:53012
I20260621 07:47:45.542912 8817 tablet_replica.cc:330] T 730c29b74a4d40c4a34c32f18ac25952 P 2c35196c6d7d417b936d34cd9d912d4e: stopping tablet replica
I20260621 07:47:45.542919 8413 tablet_service.cc:1514] Processing DeleteTablet for tablet 730c29b74a4d40c4a34c32f18ac25952 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:47:45 UTC) from {username='oryx', principal='oryx/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:45632
I20260621 07:47:45.543025 8817 raft_consensus.cc:2229] T 730c29b74a4d40c4a34c32f18ac25952 P 2c35196c6d7d417b936d34cd9d912d4e [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:47:45.543057 8818 tablet_replica.cc:330] T 730c29b74a4d40c4a34c32f18ac25952 P 0f20308d9e324531a57e487bd730ba8f: stopping tablet replica
I20260621 07:47:45.543152 8818 raft_consensus.cc:2229] T 730c29b74a4d40c4a34c32f18ac25952 P 0f20308d9e324531a57e487bd730ba8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:47:45.543228 8817 raft_consensus.cc:2258] T 730c29b74a4d40c4a34c32f18ac25952 P 2c35196c6d7d417b936d34cd9d912d4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:47:45.543247 8818 raft_consensus.cc:2258] T 730c29b74a4d40c4a34c32f18ac25952 P 0f20308d9e324531a57e487bd730ba8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:47:45.543380 8816 tablet_replica.cc:330] T 730c29b74a4d40c4a34c32f18ac25952 P 73148a30e9e54a29b9ab2894b5f3f8ae: stopping tablet replica
I20260621 07:47:45.543468 8816 raft_consensus.cc:2229] T 730c29b74a4d40c4a34c32f18ac25952 P 73148a30e9e54a29b9ab2894b5f3f8ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:47:45.543442 8282 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46498:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260621 07:47:45.543576 8816 raft_consensus.cc:2258] T 730c29b74a4d40c4a34c32f18ac25952 P 73148a30e9e54a29b9ab2894b5f3f8ae [term 1 FOLLOWER]: Raft consensus is shut down!
W20260621 07:47:45.543741 8282 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:47:45.544469 8818 ts_tablet_manager.cc:1905] T 730c29b74a4d40c4a34c32f18ac25952 P 0f20308d9e324531a57e487bd730ba8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:47:45.544718 8817 ts_tablet_manager.cc:1905] T 730c29b74a4d40c4a34c32f18ac25952 P 2c35196c6d7d417b936d34cd9d912d4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:47:45.546799 8818 ts_tablet_manager.cc:1918] T 730c29b74a4d40c4a34c32f18ac25952 P 0f20308d9e324531a57e487bd730ba8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:47:45.546859 8818 log.cc:1196] T 730c29b74a4d40c4a34c32f18ac25952 P 0f20308d9e324531a57e487bd730ba8f: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-1/wal/wals/730c29b74a4d40c4a34c32f18ac25952
I20260621 07:47:45.547119 8818 ts_tablet_manager.cc:1939] T 730c29b74a4d40c4a34c32f18ac25952 P 0f20308d9e324531a57e487bd730ba8f: Deleting consensus metadata
I20260621 07:47:45.547698 8266 catalog_manager.cc:4841] TS 0f20308d9e324531a57e487bd730ba8f (127.0.192.194:43285): tablet 730c29b74a4d40c4a34c32f18ac25952 (table test-table [id=55674ce587e84b10a7c37be75b542bbd]) successfully deleted
I20260621 07:47:45.548149 8817 ts_tablet_manager.cc:1918] T 730c29b74a4d40c4a34c32f18ac25952 P 2c35196c6d7d417b936d34cd9d912d4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:47:45.548218 8817 log.cc:1196] T 730c29b74a4d40c4a34c32f18ac25952 P 2c35196c6d7d417b936d34cd9d912d4e: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-2/wal/wals/730c29b74a4d40c4a34c32f18ac25952
I20260621 07:47:45.548483 8817 ts_tablet_manager.cc:1939] T 730c29b74a4d40c4a34c32f18ac25952 P 2c35196c6d7d417b936d34cd9d912d4e: Deleting consensus metadata
I20260621 07:47:45.549113 8267 catalog_manager.cc:4841] TS 2c35196c6d7d417b936d34cd9d912d4e (127.0.192.195:44737): tablet 730c29b74a4d40c4a34c32f18ac25952 (table test-table [id=55674ce587e84b10a7c37be75b542bbd]) successfully deleted
I20260621 07:47:45.549299 8816 ts_tablet_manager.cc:1905] T 730c29b74a4d40c4a34c32f18ac25952 P 73148a30e9e54a29b9ab2894b5f3f8ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:47:45.549691 8699 tablet_service.cc:1467] Processing CreateTablet for tablet b36b4fd580e845c48510f9ee5cf6ab01 (DEFAULT_TABLE table=test-table [id=8453fc27300d4e438d12116e01a0f901]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:47:45.549957 8699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b36b4fd580e845c48510f9ee5cf6ab01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:45.551368 8816 ts_tablet_manager.cc:1918] T 730c29b74a4d40c4a34c32f18ac25952 P 73148a30e9e54a29b9ab2894b5f3f8ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:47:45.551455 8816 log.cc:1196] T 730c29b74a4d40c4a34c32f18ac25952 P 73148a30e9e54a29b9ab2894b5f3f8ae: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-0/wal/wals/730c29b74a4d40c4a34c32f18ac25952
I20260621 07:47:45.551518 8794 tablet_bootstrap.cc:492] T b36b4fd580e845c48510f9ee5cf6ab01 P 2c35196c6d7d417b936d34cd9d912d4e: Bootstrap starting.
I20260621 07:47:45.551676 8816 ts_tablet_manager.cc:1939] T 730c29b74a4d40c4a34c32f18ac25952 P 73148a30e9e54a29b9ab2894b5f3f8ae: Deleting consensus metadata
I20260621 07:47:45.552176 8794 tablet_bootstrap.cc:654] T b36b4fd580e845c48510f9ee5cf6ab01 P 2c35196c6d7d417b936d34cd9d912d4e: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:45.552274 8266 catalog_manager.cc:4841] TS 73148a30e9e54a29b9ab2894b5f3f8ae (127.0.192.193:33397): tablet 730c29b74a4d40c4a34c32f18ac25952 (table test-table [id=55674ce587e84b10a7c37be75b542bbd]) successfully deleted
I20260621 07:47:45.552644 8559 tablet_service.cc:1467] Processing CreateTablet for tablet b36b4fd580e845c48510f9ee5cf6ab01 (DEFAULT_TABLE table=test-table [id=8453fc27300d4e438d12116e01a0f901]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:47:45.552871 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b36b4fd580e845c48510f9ee5cf6ab01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:45.553164 8794 tablet_bootstrap.cc:492] T b36b4fd580e845c48510f9ee5cf6ab01 P 2c35196c6d7d417b936d34cd9d912d4e: No bootstrap required, opened a new log
I20260621 07:47:45.553259 8794 ts_tablet_manager.cc:1397] T b36b4fd580e845c48510f9ee5cf6ab01 P 2c35196c6d7d417b936d34cd9d912d4e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260621 07:47:45.553630 8794 raft_consensus.cc:348] T b36b4fd580e845c48510f9ee5cf6ab01 P 2c35196c6d7d417b936d34cd9d912d4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } }
I20260621 07:47:45.553732 8794 raft_consensus.cc:374] T b36b4fd580e845c48510f9ee5cf6ab01 P 2c35196c6d7d417b936d34cd9d912d4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:45.553771 8794 raft_consensus.cc:729] T b36b4fd580e845c48510f9ee5cf6ab01 P 2c35196c6d7d417b936d34cd9d912d4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c35196c6d7d417b936d34cd9d912d4e, State: Initialized, Role: FOLLOWER
I20260621 07:47:45.553898 8794 consensus_queue.cc:260] T b36b4fd580e845c48510f9ee5cf6ab01 P 2c35196c6d7d417b936d34cd9d912d4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } }
I20260621 07:47:45.554136 8794 ts_tablet_manager.cc:1428] T b36b4fd580e845c48510f9ee5cf6ab01 P 2c35196c6d7d417b936d34cd9d912d4e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260621 07:47:45.554750 8793 tablet_bootstrap.cc:492] T b36b4fd580e845c48510f9ee5cf6ab01 P 0f20308d9e324531a57e487bd730ba8f: Bootstrap starting.
I20260621 07:47:45.555450 8793 tablet_bootstrap.cc:654] T b36b4fd580e845c48510f9ee5cf6ab01 P 0f20308d9e324531a57e487bd730ba8f: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:45.556174 8413 tablet_service.cc:1467] Processing CreateTablet for tablet b36b4fd580e845c48510f9ee5cf6ab01 (DEFAULT_TABLE table=test-table [id=8453fc27300d4e438d12116e01a0f901]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:47:45.556384 8413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b36b4fd580e845c48510f9ee5cf6ab01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:45.558310 8795 tablet_bootstrap.cc:492] T b36b4fd580e845c48510f9ee5cf6ab01 P 73148a30e9e54a29b9ab2894b5f3f8ae: Bootstrap starting.
I20260621 07:47:45.558609 8793 tablet_bootstrap.cc:492] T b36b4fd580e845c48510f9ee5cf6ab01 P 0f20308d9e324531a57e487bd730ba8f: No bootstrap required, opened a new log
I20260621 07:47:45.558673 8793 ts_tablet_manager.cc:1397] T b36b4fd580e845c48510f9ee5cf6ab01 P 0f20308d9e324531a57e487bd730ba8f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260621 07:47:45.558935 8795 tablet_bootstrap.cc:654] T b36b4fd580e845c48510f9ee5cf6ab01 P 73148a30e9e54a29b9ab2894b5f3f8ae: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:45.558993 8793 raft_consensus.cc:348] T b36b4fd580e845c48510f9ee5cf6ab01 P 0f20308d9e324531a57e487bd730ba8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } }
I20260621 07:47:45.559083 8793 raft_consensus.cc:374] T b36b4fd580e845c48510f9ee5cf6ab01 P 0f20308d9e324531a57e487bd730ba8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:45.559113 8793 raft_consensus.cc:729] T b36b4fd580e845c48510f9ee5cf6ab01 P 0f20308d9e324531a57e487bd730ba8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f20308d9e324531a57e487bd730ba8f, State: Initialized, Role: FOLLOWER
I20260621 07:47:45.559218 8793 consensus_queue.cc:260] T b36b4fd580e845c48510f9ee5cf6ab01 P 0f20308d9e324531a57e487bd730ba8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } }
I20260621 07:47:45.559444 8793 ts_tablet_manager.cc:1428] T b36b4fd580e845c48510f9ee5cf6ab01 P 0f20308d9e324531a57e487bd730ba8f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260621 07:47:45.559895 8795 tablet_bootstrap.cc:492] T b36b4fd580e845c48510f9ee5cf6ab01 P 73148a30e9e54a29b9ab2894b5f3f8ae: No bootstrap required, opened a new log
I20260621 07:47:45.559988 8795 ts_tablet_manager.cc:1397] T b36b4fd580e845c48510f9ee5cf6ab01 P 73148a30e9e54a29b9ab2894b5f3f8ae: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260621 07:47:45.560400 8795 raft_consensus.cc:348] T b36b4fd580e845c48510f9ee5cf6ab01 P 73148a30e9e54a29b9ab2894b5f3f8ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } }
I20260621 07:47:45.560488 8795 raft_consensus.cc:374] T b36b4fd580e845c48510f9ee5cf6ab01 P 73148a30e9e54a29b9ab2894b5f3f8ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:45.560510 8795 raft_consensus.cc:729] T b36b4fd580e845c48510f9ee5cf6ab01 P 73148a30e9e54a29b9ab2894b5f3f8ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73148a30e9e54a29b9ab2894b5f3f8ae, State: Initialized, Role: FOLLOWER
I20260621 07:47:45.560647 8795 consensus_queue.cc:260] T b36b4fd580e845c48510f9ee5cf6ab01 P 73148a30e9e54a29b9ab2894b5f3f8ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } }
I20260621 07:47:45.560940 8795 ts_tablet_manager.cc:1428] T b36b4fd580e845c48510f9ee5cf6ab01 P 73148a30e9e54a29b9ab2894b5f3f8ae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260621 07:47:45.676168 8834 raft_consensus.cc:482] T b36b4fd580e845c48510f9ee5cf6ab01 P 73148a30e9e54a29b9ab2894b5f3f8ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:47:45.676343 8834 raft_consensus.cc:504] T b36b4fd580e845c48510f9ee5cf6ab01 P 73148a30e9e54a29b9ab2894b5f3f8ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } }
I20260621 07:47:45.676815 8834 leader_election.cc:288] T b36b4fd580e845c48510f9ee5cf6ab01 P 73148a30e9e54a29b9ab2894b5f3f8ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f20308d9e324531a57e487bd730ba8f (127.0.192.194:43285), 2c35196c6d7d417b936d34cd9d912d4e (127.0.192.195:44737)
I20260621 07:47:45.677254 8579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b36b4fd580e845c48510f9ee5cf6ab01" candidate_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f20308d9e324531a57e487bd730ba8f" is_pre_election: true
I20260621 07:47:45.677402 8579 raft_consensus.cc:2454] T b36b4fd580e845c48510f9ee5cf6ab01 P 0f20308d9e324531a57e487bd730ba8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73148a30e9e54a29b9ab2894b5f3f8ae in term 0.
I20260621 07:47:45.677757 8370 leader_election.cc:302] T b36b4fd580e845c48510f9ee5cf6ab01 P 73148a30e9e54a29b9ab2894b5f3f8ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f20308d9e324531a57e487bd730ba8f, 73148a30e9e54a29b9ab2894b5f3f8ae; no voters:
I20260621 07:47:45.677928 8834 raft_consensus.cc:2781] T b36b4fd580e845c48510f9ee5cf6ab01 P 73148a30e9e54a29b9ab2894b5f3f8ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:47:45.677976 8834 raft_consensus.cc:482] T b36b4fd580e845c48510f9ee5cf6ab01 P 73148a30e9e54a29b9ab2894b5f3f8ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:47:45.678013 8834 raft_consensus.cc:3037] T b36b4fd580e845c48510f9ee5cf6ab01 P 73148a30e9e54a29b9ab2894b5f3f8ae [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:45.678058 8719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b36b4fd580e845c48510f9ee5cf6ab01" candidate_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c35196c6d7d417b936d34cd9d912d4e" is_pre_election: true
I20260621 07:47:45.678185 8719 raft_consensus.cc:2454] T b36b4fd580e845c48510f9ee5cf6ab01 P 2c35196c6d7d417b936d34cd9d912d4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73148a30e9e54a29b9ab2894b5f3f8ae in term 0.
I20260621 07:47:45.679391 8834 raft_consensus.cc:504] T b36b4fd580e845c48510f9ee5cf6ab01 P 73148a30e9e54a29b9ab2894b5f3f8ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } }
I20260621 07:47:45.679755 8834 leader_election.cc:288] T b36b4fd580e845c48510f9ee5cf6ab01 P 73148a30e9e54a29b9ab2894b5f3f8ae [CANDIDATE]: Term 1 election: Requested vote from peers 0f20308d9e324531a57e487bd730ba8f (127.0.192.194:43285), 2c35196c6d7d417b936d34cd9d912d4e (127.0.192.195:44737)
I20260621 07:47:45.680400 8579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b36b4fd580e845c48510f9ee5cf6ab01" candidate_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f20308d9e324531a57e487bd730ba8f"
I20260621 07:47:45.680523 8579 raft_consensus.cc:3037] T b36b4fd580e845c48510f9ee5cf6ab01 P 0f20308d9e324531a57e487bd730ba8f [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:45.681143 8719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b36b4fd580e845c48510f9ee5cf6ab01" candidate_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c35196c6d7d417b936d34cd9d912d4e"
I20260621 07:47:45.681234 8719 raft_consensus.cc:3037] T b36b4fd580e845c48510f9ee5cf6ab01 P 2c35196c6d7d417b936d34cd9d912d4e [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:45.681823 8579 raft_consensus.cc:2454] T b36b4fd580e845c48510f9ee5cf6ab01 P 0f20308d9e324531a57e487bd730ba8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73148a30e9e54a29b9ab2894b5f3f8ae in term 1.
I20260621 07:47:45.681978 8719 raft_consensus.cc:2454] T b36b4fd580e845c48510f9ee5cf6ab01 P 2c35196c6d7d417b936d34cd9d912d4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73148a30e9e54a29b9ab2894b5f3f8ae in term 1.
I20260621 07:47:45.682219 8370 leader_election.cc:302] T b36b4fd580e845c48510f9ee5cf6ab01 P 73148a30e9e54a29b9ab2894b5f3f8ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f20308d9e324531a57e487bd730ba8f, 73148a30e9e54a29b9ab2894b5f3f8ae; no voters:
I20260621 07:47:45.682384 8834 raft_consensus.cc:2781] T b36b4fd580e845c48510f9ee5cf6ab01 P 73148a30e9e54a29b9ab2894b5f3f8ae [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:47:45.682512 8834 raft_consensus.cc:686] T b36b4fd580e845c48510f9ee5cf6ab01 P 73148a30e9e54a29b9ab2894b5f3f8ae [term 1 LEADER]: Becoming Leader. State: Replica: 73148a30e9e54a29b9ab2894b5f3f8ae, State: Running, Role: LEADER
I20260621 07:47:45.682636 8834 consensus_queue.cc:237] T b36b4fd580e845c48510f9ee5cf6ab01 P 73148a30e9e54a29b9ab2894b5f3f8ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } }
I20260621 07:47:45.684098 8280 catalog_manager.cc:5495] T b36b4fd580e845c48510f9ee5cf6ab01 P 73148a30e9e54a29b9ab2894b5f3f8ae reported cstate change: term changed from 0 to 1, leader changed from <none> to 73148a30e9e54a29b9ab2894b5f3f8ae (127.0.192.193). New cstate: current_term: 1 leader_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73148a30e9e54a29b9ab2894b5f3f8ae" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 33397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 } health_report { overall_health: UNKNOWN } } }
I20260621 07:47:45.721333 8792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:45.718098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45656 (local address 127.0.192.193:33397)
0621 07:47:45.718185 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:45.718189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:45.718212 (+ 23us) server_negotiation.cc:407] Connection header received
0621 07:47:45.718369 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:45.718371 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:45.718416 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:45.718493 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:47:45.718776 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:45.719263 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:45.720362 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:45.720614 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:45.720701 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:45.720970 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:47:45.721032 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:47:45.721079 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:45.721182 (+ 103us) server_negotiation.cc:299] Negotiation successful
0621 07:47:45.721220 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260621 07:47:45.729883 8792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:45.726566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:35179 (local address 127.0.192.193:33397)
0621 07:47:45.726692 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:45.726696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:45.726773 (+ 77us) server_negotiation.cc:407] Connection header received
0621 07:47:45.726880 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:45.726881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:45.726915 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:45.727025 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:47:45.727279 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:45.727750 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:45.728949 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:45.729152 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:45.729232 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:45.729490 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:47:45.729548 (+ 58us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:47:45.729596 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:45.729718 (+ 122us) server_negotiation.cc:299] Negotiation successful
0621 07:47:45.729759 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 07:47:45.730015 8880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:45.726473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:33397 (local address 127.0.192.193:35179)
0621 07:47:45.726690 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:45.726705 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:45.726798 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:45.727036 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:45.727039 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:45.727047 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 07:47:45.727159 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:45.727165 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:45.727863 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:45.727867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:45.728836 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:45.728846 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:45.729257 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:45.729261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:45.729346 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:45.729377 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 07:47:45.729623 (+ 246us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 07:47:45.729629 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 07:47:45.729708 (+ 79us) client_negotiation.cc:240] Negotiation successful
0621 07:47:45.729809 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":4,"thread_start_us":73,"threads_started":1}
I20260621 07:47:45.735085 8579 raft_consensus.cc:1261] T b36b4fd580e845c48510f9ee5cf6ab01 P 0f20308d9e324531a57e487bd730ba8f [term 1 FOLLOWER]: Refusing update from remote peer 73148a30e9e54a29b9ab2894b5f3f8ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:47:45.735123 8719 raft_consensus.cc:1261] T b36b4fd580e845c48510f9ee5cf6ab01 P 2c35196c6d7d417b936d34cd9d912d4e [term 1 FOLLOWER]: Refusing update from remote peer 73148a30e9e54a29b9ab2894b5f3f8ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:47:45.735478 8801 consensus_queue.cc:1035] T b36b4fd580e845c48510f9ee5cf6ab01 P 73148a30e9e54a29b9ab2894b5f3f8ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c35196c6d7d417b936d34cd9d912d4e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:47:45.735641 8834 consensus_queue.cc:1035] T b36b4fd580e845c48510f9ee5cf6ab01 P 73148a30e9e54a29b9ab2894b5f3f8ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f20308d9e324531a57e487bd730ba8f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:47:45.767076 8280 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:47:45.767237 8280 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:47:45.769132 8280 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a5a7e6a80b74aebbeb361add74240ab: Sending DeleteTablet for 3 replicas of tablet b36b4fd580e845c48510f9ee5cf6ab01
I20260621 07:47:45.769690 8413 tablet_service.cc:1514] Processing DeleteTablet for tablet b36b4fd580e845c48510f9ee5cf6ab01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:47:45 UTC) from {username='oryx', principal='oryx/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:45632
I20260621 07:47:45.769846 8816 tablet_replica.cc:330] T b36b4fd580e845c48510f9ee5cf6ab01 P 73148a30e9e54a29b9ab2894b5f3f8ae: stopping tablet replica
I20260621 07:47:45.769829 8559 tablet_service.cc:1514] Processing DeleteTablet for tablet b36b4fd580e845c48510f9ee5cf6ab01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:47:45 UTC) from {username='oryx', principal='oryx/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:53012
I20260621 07:47:45.769985 8816 raft_consensus.cc:2229] T b36b4fd580e845c48510f9ee5cf6ab01 P 73148a30e9e54a29b9ab2894b5f3f8ae [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:47:45.770045 8818 tablet_replica.cc:330] T b36b4fd580e845c48510f9ee5cf6ab01 P 0f20308d9e324531a57e487bd730ba8f: stopping tablet replica
I20260621 07:47:45.770047 8699 tablet_service.cc:1514] Processing DeleteTablet for tablet b36b4fd580e845c48510f9ee5cf6ab01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:47:45 UTC) from {username='oryx', principal='oryx/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:58036
I20260621 07:47:45.770155 8818 raft_consensus.cc:2229] T b36b4fd580e845c48510f9ee5cf6ab01 P 0f20308d9e324531a57e487bd730ba8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:47:45.770195 8817 tablet_replica.cc:330] T b36b4fd580e845c48510f9ee5cf6ab01 P 2c35196c6d7d417b936d34cd9d912d4e: stopping tablet replica
I20260621 07:47:45.770279 8818 raft_consensus.cc:2258] T b36b4fd580e845c48510f9ee5cf6ab01 P 0f20308d9e324531a57e487bd730ba8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:47:45.770272 8816 raft_consensus.cc:2258] T b36b4fd580e845c48510f9ee5cf6ab01 P 73148a30e9e54a29b9ab2894b5f3f8ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:47:45.770284 8817 raft_consensus.cc:2229] T b36b4fd580e845c48510f9ee5cf6ab01 P 2c35196c6d7d417b936d34cd9d912d4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:47:45.770373 8817 raft_consensus.cc:2258] T b36b4fd580e845c48510f9ee5cf6ab01 P 2c35196c6d7d417b936d34cd9d912d4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:47:45.771083 8817 ts_tablet_manager.cc:1905] T b36b4fd580e845c48510f9ee5cf6ab01 P 2c35196c6d7d417b936d34cd9d912d4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:47:45.771355 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 8354
I20260621 07:47:45.771358 8816 ts_tablet_manager.cc:1905] T b36b4fd580e845c48510f9ee5cf6ab01 P 73148a30e9e54a29b9ab2894b5f3f8ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:47:45.773995 8818 ts_tablet_manager.cc:1905] T b36b4fd580e845c48510f9ee5cf6ab01 P 0f20308d9e324531a57e487bd730ba8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:47:45.774166 8817 ts_tablet_manager.cc:1918] T b36b4fd580e845c48510f9ee5cf6ab01 P 2c35196c6d7d417b936d34cd9d912d4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 07:47:45.774257 8817 log.cc:1196] T b36b4fd580e845c48510f9ee5cf6ab01 P 2c35196c6d7d417b936d34cd9d912d4e: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-2/wal/wals/b36b4fd580e845c48510f9ee5cf6ab01
I20260621 07:47:45.774487 8817 ts_tablet_manager.cc:1939] T b36b4fd580e845c48510f9ee5cf6ab01 P 2c35196c6d7d417b936d34cd9d912d4e: Deleting consensus metadata
I20260621 07:47:45.775017 8267 catalog_manager.cc:4841] TS 2c35196c6d7d417b936d34cd9d912d4e (127.0.192.195:44737): tablet b36b4fd580e845c48510f9ee5cf6ab01 (table test-table [id=8453fc27300d4e438d12116e01a0f901]) successfully deleted
I20260621 07:47:45.776996 8818 ts_tablet_manager.cc:1918] T b36b4fd580e845c48510f9ee5cf6ab01 P 0f20308d9e324531a57e487bd730ba8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 07:47:45.777083 8818 log.cc:1196] T b36b4fd580e845c48510f9ee5cf6ab01 P 0f20308d9e324531a57e487bd730ba8f: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782028028522580-771-0/minicluster-data/ts-1/wal/wals/b36b4fd580e845c48510f9ee5cf6ab01
I20260621 07:47:45.777328 8818 ts_tablet_manager.cc:1939] T b36b4fd580e845c48510f9ee5cf6ab01 P 0f20308d9e324531a57e487bd730ba8f: Deleting consensus metadata
I20260621 07:47:45.777940 8266 catalog_manager.cc:4841] TS 0f20308d9e324531a57e487bd730ba8f (127.0.192.194:43285): tablet b36b4fd580e845c48510f9ee5cf6ab01 (table test-table [id=8453fc27300d4e438d12116e01a0f901]) successfully deleted
W20260621 07:47:45.780906 8266 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.192.193:33397 (error 108)
I20260621 07:47:45.781608 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 8494
I20260621 07:47:45.781874 8888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:45.781469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:33397 (local address 127.0.0.1:45662)
0621 07:47:45.781685 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:45.781762 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.193:33397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":60,"threads_started":1}
W20260621 07:47:45.782225 8266 catalog_manager.cc:4584] TS 73148a30e9e54a29b9ab2894b5f3f8ae (127.0.192.193:33397): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b36b4fd580e845c48510f9ee5cf6ab01: Network error: Client connection negotiation failed: client connection to 127.0.192.193:33397: connect: Connection refused (error 111)
I20260621 07:47:45.788226 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 8634
I20260621 07:47:45.794485 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 8250
2026-06-21T07:47:45Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2988 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782028028522580-771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:47:45 dist-test-slave-lzx3 krb5kdc[8891](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:47:45 dist-test-slave-lzx3 krb5kdc[8891](info): set up 2 sockets
Jun 21 07:47:45 dist-test-slave-lzx3 krb5kdc[8891](info): commencing operation
krb5kdc: starting...
W20260621 07:47:46.788990 771 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782028028522580-771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782028028522580-771-0/krb5kdc/test-user.keytab.
Jun 21 07:47:46 dist-test-slave-lzx3 krb5kdc[8891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:47:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:47:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.192.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
I20260621 07:47:46.893540 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:43733
--webserver_interface=127.0.192.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:36095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:43733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:46.995450 8904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:46.995688 8904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:46.995725 8904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:46.999482 8904 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:47:46.999527 8904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:46.999547 8904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:46.999562 8904 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:47:46.999576 8904 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:47:47.004365 8904 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:36095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:43733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:43733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:47.005234 8904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:47.006059 8904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:47.013659 8909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:47.013731 8912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:47.013789 8904 server_base.cc:1034] running on GCE node
W20260621 07:47:47.013687 8910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:47.014377 8904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:47.015342 8904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:47.016502 8904 hybrid_clock.cc:648] HybridClock initialized: now 1782028067016489 us; error 24 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-21T07:47:47Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1224 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:47:47 dist-test-slave-lzx3 krb5kdc[8917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:47:47 dist-test-slave-lzx3 krb5kdc[8917](info): set up 2 sockets
Jun 21 07:47:47 dist-test-slave-lzx3 krb5kdc[8917](info): commencing operation
krb5kdc: starting...
W20260621 07:47:47.988969 771 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/test-user.keytab.
Jun 21 07:47:48 dist-test-slave-lzx3 krb5kdc[8917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:47:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:47:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.192.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
I20260621 07:47:48.097347 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:42251
--webserver_interface=127.0.192.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:36829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:42251
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:48.196153 8930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:48.196368 8930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:48.196406 8930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:48.200206 8930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:47:48.200253 8930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:48.200270 8930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:48.200323 8930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:47:48.200347 8930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:47:48.204844 8930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:36829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:42251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:42251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:48.205652 8930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:48.206498 8930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:48.213634 8936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:48.213629 8935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:48.213629 8938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:48.213719 8930 server_base.cc:1034] running on GCE node
I20260621 07:47:48.214349 8930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:48.215260 8930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:48.216459 8930 hybrid_clock.cc:648] HybridClock initialized: now 1782028068216436 us; error 34 us; skew 500 ppm
Jun 21 07:47:48 dist-test-slave-lzx3 krb5kdc[8917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028068, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:48.222086 8930 init.cc:377] Logged in from keytab as kudu/127.0.192.254@KRBTEST.COM (short username kudu)
I20260621 07:47:48.222867 8930 webserver.cc:460] Webserver started at http://127.0.192.254:46565/ using document root <none> and password file <none>
I20260621 07:47:48.223322 8930 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:48.223407 8930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:48.223618 8930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:48.225450 8930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "dab035a0a5b146879905c833b365561e"
format_stamp: "Formatted at 2026-06-21 07:47:48 on dist-test-slave-lzx3"
server_key: "b8fedb78780bcae9f7c7da26d4bbe983"
server_key_iv: "75571f8492e208842abdef45e3ba5582"
server_key_version: "encryptionkey@0"
I20260621 07:47:48.225971 8930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "dab035a0a5b146879905c833b365561e"
format_stamp: "Formatted at 2026-06-21 07:47:48 on dist-test-slave-lzx3"
server_key: "b8fedb78780bcae9f7c7da26d4bbe983"
server_key_iv: "75571f8492e208842abdef45e3ba5582"
server_key_version: "encryptionkey@0"
I20260621 07:47:48.229223 8930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:47:48.231385 8945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:48.232461 8930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:48.232548 8930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "dab035a0a5b146879905c833b365561e"
format_stamp: "Formatted at 2026-06-21 07:47:48 on dist-test-slave-lzx3"
server_key: "b8fedb78780bcae9f7c7da26d4bbe983"
server_key_iv: "75571f8492e208842abdef45e3ba5582"
server_key_version: "encryptionkey@0"
I20260621 07:47:48.232677 8930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:48.247262 8930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:48.249742 8930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:48.249879 8930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:48.256421 8930 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:42251
I20260621 07:47:48.256431 8997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:42251 every 8 connection(s)
I20260621 07:47:48.257275 8930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:47:48.259748 8998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:47:48.261986 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 8930
I20260621 07:47:48.262159 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:47:48.262431 771 external_mini_cluster.cc:1428] Setting key 92d4f1525221e0c3ddedf00cfe91c3a9
I20260621 07:47:48.264945 8998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dab035a0a5b146879905c833b365561e: Bootstrap starting.
I20260621 07:47:48.266764 8998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dab035a0a5b146879905c833b365561e: Neither blocks nor log segments found. Creating new log.
I20260621 07:47:48.267395 8998 log.cc:824] T 00000000000000000000000000000000 P dab035a0a5b146879905c833b365561e: Log is configured to *not* fsync() on all Append() calls
I20260621 07:47:48.268976 8998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dab035a0a5b146879905c833b365561e: No bootstrap required, opened a new log
Jun 21 07:47:48 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:47:48.272406 8998 raft_consensus.cc:348] T 00000000000000000000000000000000 P dab035a0a5b146879905c833b365561e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab035a0a5b146879905c833b365561e" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 42251 } }
I20260621 07:47:48.272677 8998 raft_consensus.cc:374] T 00000000000000000000000000000000 P dab035a0a5b146879905c833b365561e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:47:48.272791 8998 raft_consensus.cc:729] T 00000000000000000000000000000000 P dab035a0a5b146879905c833b365561e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dab035a0a5b146879905c833b365561e, State: Initialized, Role: FOLLOWER
I20260621 07:47:48.273254 8998 consensus_queue.cc:260] T 00000000000000000000000000000000 P dab035a0a5b146879905c833b365561e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab035a0a5b146879905c833b365561e" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 42251 } }
I20260621 07:47:48.273379 8998 raft_consensus.cc:388] T 00000000000000000000000000000000 P dab035a0a5b146879905c833b365561e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:47:48.273448 8998 raft_consensus.cc:482] T 00000000000000000000000000000000 P dab035a0a5b146879905c833b365561e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:47:48.273559 8998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dab035a0a5b146879905c833b365561e [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:47:48.274511 8998 raft_consensus.cc:504] T 00000000000000000000000000000000 P dab035a0a5b146879905c833b365561e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab035a0a5b146879905c833b365561e" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 42251 } }
I20260621 07:47:48.274834 8998 leader_election.cc:302] T 00000000000000000000000000000000 P dab035a0a5b146879905c833b365561e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dab035a0a5b146879905c833b365561e; no voters:
I20260621 07:47:48.275086 8998 leader_election.cc:288] T 00000000000000000000000000000000 P dab035a0a5b146879905c833b365561e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:47:48.275202 9001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:48.263566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52028 (local address 127.0.192.254:42251)
0621 07:47:48.263911 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:48.263924 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:48.263954 (+ 30us) server_negotiation.cc:407] Connection header received
0621 07:47:48.264488 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:48.264502 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:48.264847 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:48.265147 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:48.265860 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:48.266721 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:48.267538 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:48.267949 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:48.268101 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:48.270679 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:48.270683 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:48.270694 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:48.270716 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:48.272286 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:48.272769 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:48.272774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:48.272779 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:48.272923 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:48.273199 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:48.273201 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:48.273202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:48.273700 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:48.273868 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:48.274124 (+ 256us) server_negotiation.cc:299] Negotiation successful
0621 07:47:48.274386 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":112,"threads_started":1}
I20260621 07:47:48.275208 9003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dab035a0a5b146879905c833b365561e [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:47:48.275421 9003 raft_consensus.cc:686] T 00000000000000000000000000000000 P dab035a0a5b146879905c833b365561e [term 1 LEADER]: Becoming Leader. State: Replica: dab035a0a5b146879905c833b365561e, State: Running, Role: LEADER
I20260621 07:47:48.275761 9003 consensus_queue.cc:237] T 00000000000000000000000000000000 P dab035a0a5b146879905c833b365561e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab035a0a5b146879905c833b365561e" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 42251 } }
I20260621 07:47:48.276201 8998 sys_catalog.cc:564] T 00000000000000000000000000000000 P dab035a0a5b146879905c833b365561e [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:47:48.276834 9005 sys_catalog.cc:455] T 00000000000000000000000000000000 P dab035a0a5b146879905c833b365561e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dab035a0a5b146879905c833b365561e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab035a0a5b146879905c833b365561e" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 42251 } } }
I20260621 07:47:48.276834 9003 sys_catalog.cc:455] T 00000000000000000000000000000000 P dab035a0a5b146879905c833b365561e [sys.catalog]: SysCatalogTable state changed. Reason: New leader dab035a0a5b146879905c833b365561e. Latest consensus state: current_term: 1 leader_uuid: "dab035a0a5b146879905c833b365561e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab035a0a5b146879905c833b365561e" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 42251 } } }
I20260621 07:47:48.276988 9003 sys_catalog.cc:458] T 00000000000000000000000000000000 P dab035a0a5b146879905c833b365561e [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:48.276988 9005 sys_catalog.cc:458] T 00000000000000000000000000000000 P dab035a0a5b146879905c833b365561e [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:48.277482 9011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:47:48.280128 9011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:47:48.285008 9011 catalog_manager.cc:1347] Generated new cluster ID: 3b3386dc31d041d0a57b9bc607730f28
I20260621 07:47:48.285060 9011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:47:48.312965 9011 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:47:48.313742 9011 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:47:48.320151 9011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dab035a0a5b146879905c833b365561e: Generated new TSK 0
I20260621 07:47:48.320735 9011 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.192.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab.
Entry for principal kudu/127.0.192.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab.
I20260621 07:47:48.347563 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.193:0
--local_ip_for_outbound_sockets=127.0.192.193
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:42251
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:36829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:48.462553 9024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:48.462761 9024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:48.462797 9024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:48.466115 9024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:48.466157 9024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:48.466207 9024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.193
I20260621 07:47:48.470904 9024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:36829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:42251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.193
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:48.471692 9024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:48.472573 9024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:48.481266 9032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:48.481369 9029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:48.481451 9024 server_base.cc:1034] running on GCE node
W20260621 07:47:48.481273 9030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:48.481901 9024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:48.482436 9024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:48.483609 9024 hybrid_clock.cc:648] HybridClock initialized: now 1782028068483598 us; error 21 us; skew 500 ppm
Jun 21 07:47:48 dist-test-slave-lzx3 krb5kdc[8917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028068, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:48.486045 9024 init.cc:377] Logged in from keytab as kudu/127.0.192.193@KRBTEST.COM (short username kudu)
I20260621 07:47:48.486739 9024 webserver.cc:460] Webserver started at http://127.0.192.193:43403/ using document root <none> and password file <none>
I20260621 07:47:48.487178 9024 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:48.487228 9024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:48.487365 9024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:48.489123 9024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-0/data/instance:
uuid: "9f3cd4138e8d4a1cbc9d01b7dd5c13a8"
format_stamp: "Formatted at 2026-06-21 07:47:48 on dist-test-slave-lzx3"
server_key: "9bdbec867d62b8d86e89d075007f165d"
server_key_iv: "a2e5be60ba03842c1ffd4fc2fda5ebb5"
server_key_version: "encryptionkey@0"
I20260621 07:47:48.489589 9024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-0/wal/instance:
uuid: "9f3cd4138e8d4a1cbc9d01b7dd5c13a8"
format_stamp: "Formatted at 2026-06-21 07:47:48 on dist-test-slave-lzx3"
server_key: "9bdbec867d62b8d86e89d075007f165d"
server_key_iv: "a2e5be60ba03842c1ffd4fc2fda5ebb5"
server_key_version: "encryptionkey@0"
I20260621 07:47:48.492671 9024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:47:48.494675 9039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:48.495837 9024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:48.495977 9024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-0/wal
uuid: "9f3cd4138e8d4a1cbc9d01b7dd5c13a8"
format_stamp: "Formatted at 2026-06-21 07:47:48 on dist-test-slave-lzx3"
server_key: "9bdbec867d62b8d86e89d075007f165d"
server_key_iv: "a2e5be60ba03842c1ffd4fc2fda5ebb5"
server_key_version: "encryptionkey@0"
I20260621 07:47:48.496073 9024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:48.513863 9024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:48.516237 9024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:48.516420 9024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:48.516963 9024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:47:48.517627 9024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:47:48.517668 9024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:48.517726 9024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:47:48.517752 9024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:48.525487 9024 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.193:36865
I20260621 07:47:48.525516 9152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.193:36865 every 8 connection(s)
I20260621 07:47:48.526203 9024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
I20260621 07:47:48.533001 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 9024
I20260621 07:47:48.533115 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-0/wal/instance
I20260621 07:47:48.533439 771 external_mini_cluster.cc:1428] Setting key b1f1c6ac574892f244a3fa5f2a553c77
Jun 21 07:47:48 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028068, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:47:48.538290 9001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:48.527678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:41781 (local address 127.0.192.254:42251)
0621 07:47:48.527797 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:48.527800 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:48.528453 (+ 653us) server_negotiation.cc:407] Connection header received
0621 07:47:48.529778 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:48.529782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:48.529841 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:48.529940 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:48.531012 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:48.531473 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:48.532430 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:48.532695 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:48.532793 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:48.535055 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:48.535058 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:48.535059 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:48.535078 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:48.536424 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:48.536930 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:48.536935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:48.536937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:48.537010 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:48.537350 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:48.537355 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:48.537356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:48.537519 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:48.537618 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:48.538033 (+ 415us) server_negotiation.cc:299] Negotiation successful
0621 07:47:48.538150 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 07:47:48.539070 9155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:48.527964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.193:41781)
0621 07:47:48.528330 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:48.528364 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:48.529572 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:48.530095 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:48.530101 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:48.530464 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:48.530888 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:48.530899 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:48.531585 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:48.531590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:48.532314 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:48.532322 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:48.532821 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:48.532824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:48.532970 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:48.533494 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:48.533516 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:48.534918 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:48.536539 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:48.536545 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:48.536573 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:48.536831 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:48.537097 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:48.537099 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:48.537101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:48.537219 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:48.537616 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:48.537622 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:48.537825 (+ 203us) client_negotiation.cc:769] Sending connection context
0621 07:47:48.537996 (+ 171us) client_negotiation.cc:240] Negotiation successful
0621 07:47:48.538277 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":94,"threads_started":1}
I20260621 07:47:48.540391 9153 heartbeater.cc:347] Connected to a master server at 127.0.192.254:42251
I20260621 07:47:48.540639 9153 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:48.541059 9153 heartbeater.cc:511] Master 127.0.192.254:42251 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.192.194@KRBTEST.COM; defaulting to no policy
I20260621 07:47:48.542341 8962 ts_manager.cc:194] Registered new tserver with Master: 9f3cd4138e8d4a1cbc9d01b7dd5c13a8 (127.0.192.193:36865)
I20260621 07:47:48.543336 8962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.193@KRBTEST.COM'} at 127.0.192.193:41781
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab.
Entry for principal kudu/127.0.192.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab.
I20260621 07:47:48.558699 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.194:0
--local_ip_for_outbound_sockets=127.0.192.194
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:42251
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:36829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:48.657258 9158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:48.657465 9158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:48.657500 9158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:48.660987 9158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:48.661031 9158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:48.661098 9158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.194
I20260621 07:47:48.665500 9158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:36829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:42251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.194
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:48.666244 9158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:48.667064 9158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:48.675084 9163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:48.675200 9158 server_base.cc:1034] running on GCE node
W20260621 07:47:48.675117 9166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:48.675151 9164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:48.675808 9158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:48.676386 9158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:48.677593 9158 hybrid_clock.cc:648] HybridClock initialized: now 1782028068677549 us; error 68 us; skew 500 ppm
Jun 21 07:47:48 dist-test-slave-lzx3 krb5kdc[8917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028068, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:48.680121 9158 init.cc:377] Logged in from keytab as kudu/127.0.192.194@KRBTEST.COM (short username kudu)
I20260621 07:47:48.680995 9158 webserver.cc:460] Webserver started at http://127.0.192.194:43723/ using document root <none> and password file <none>
I20260621 07:47:48.681479 9158 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:48.681560 9158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:48.681746 9158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:48.683477 9158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-1/data/instance:
uuid: "10877c6410584af4ad1167ec386c70e4"
format_stamp: "Formatted at 2026-06-21 07:47:48 on dist-test-slave-lzx3"
server_key: "c454da0f4b14279cd4b877801e84ec54"
server_key_iv: "51fde3e2c6a40f5c87a0802d55af36a3"
server_key_version: "encryptionkey@0"
I20260621 07:47:48.683982 9158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-1/wal/instance:
uuid: "10877c6410584af4ad1167ec386c70e4"
format_stamp: "Formatted at 2026-06-21 07:47:48 on dist-test-slave-lzx3"
server_key: "c454da0f4b14279cd4b877801e84ec54"
server_key_iv: "51fde3e2c6a40f5c87a0802d55af36a3"
server_key_version: "encryptionkey@0"
I20260621 07:47:48.687172 9158 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 07:47:48.689195 9173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:48.690294 9158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:48.690419 9158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-1/wal
uuid: "10877c6410584af4ad1167ec386c70e4"
format_stamp: "Formatted at 2026-06-21 07:47:48 on dist-test-slave-lzx3"
server_key: "c454da0f4b14279cd4b877801e84ec54"
server_key_iv: "51fde3e2c6a40f5c87a0802d55af36a3"
server_key_version: "encryptionkey@0"
I20260621 07:47:48.690510 9158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:48.700078 9158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:48.702322 9158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:48.702494 9158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:48.702970 9158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:47:48.703716 9158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:47:48.703783 9158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:48.703850 9158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:47:48.703896 9158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:48.712232 9158 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.194:36135
I20260621 07:47:48.712249 9286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.194:36135 every 8 connection(s)
I20260621 07:47:48.712990 9158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
I20260621 07:47:48.713603 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 9158
I20260621 07:47:48.713735 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-1/wal/instance
I20260621 07:47:48.714000 771 external_mini_cluster.cc:1428] Setting key ee7ef025613e0db6fe925daa34aec67e
Jun 21 07:47:48 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028068, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.192.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.195@KRBTEST.COM" created.
I20260621 07:47:48.726843 9001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:48.714599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:36767 (local address 127.0.192.254:42251)
0621 07:47:48.714746 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:48.714750 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:48.715569 (+ 819us) server_negotiation.cc:407] Connection header received
0621 07:47:48.716989 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:48.716992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:48.717045 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:48.717109 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:48.718396 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:48.718850 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:48.720648 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:48.720840 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:48.720918 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:48.723371 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:48.723374 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:48.723376 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:48.723395 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:48.724941 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:48.725416 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:48.725421 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:48.725423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:48.725502 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:48.725850 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:48.725854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:48.725856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:48.726032 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:48.726134 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:48.726588 (+ 454us) server_negotiation.cc:299] Negotiation successful
0621 07:47:48.726692 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 07:47:48.727463 9289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:48.714996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.194:36767)
0621 07:47:48.715426 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:48.715472 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:48.716739 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:48.717277 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:48.717286 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:48.717742 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:48.718251 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:48.718263 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:48.718976 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:48.718987 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:48.720503 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:48.720517 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:48.720991 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:48.720999 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:48.721223 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:48.721740 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:48.721761 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:48.723219 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:48.725060 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:48.725065 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:48.725077 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:48.725314 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:48.725607 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:48.725610 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:48.725612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:48.725730 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:48.726143 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:48.726150 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:48.726384 (+ 234us) client_negotiation.cc:769] Sending connection context
0621 07:47:48.726548 (+ 164us) client_negotiation.cc:240] Negotiation successful
0621 07:47:48.726805 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":126,"threads_started":1}
I20260621 07:47:48.728744 9287 heartbeater.cc:347] Connected to a master server at 127.0.192.254:42251
I20260621 07:47:48.728945 9287 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:48.729393 9287 heartbeater.cc:511] Master 127.0.192.254:42251 requested a full tablet report, sending...
I20260621 07:47:48.730403 8962 ts_manager.cc:194] Registered new tserver with Master: 10877c6410584af4ad1167ec386c70e4 (127.0.192.194:36135)
I20260621 07:47:48.731228 8962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.194@KRBTEST.COM'} at 127.0.192.194:36767
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab.
Entry for principal kudu/127.0.192.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab.
I20260621 07:47:48.740624 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.195:0
--local_ip_for_outbound_sockets=127.0.192.195
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:42251
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:36829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:48.842156 9292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:48.842368 9292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:48.842437 9292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:48.846279 9292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:48.846325 9292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:48.846423 9292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.195
I20260621 07:47:48.851279 9292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:36829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:42251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.195
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:48.852190 9292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:48.853256 9292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:48.862003 9297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:48.862025 9300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:48.862059 9292 server_base.cc:1034] running on GCE node
W20260621 07:47:48.862003 9298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:48.862705 9292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:48.863207 9292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:48.864357 9292 hybrid_clock.cc:648] HybridClock initialized: now 1782028068864341 us; error 27 us; skew 500 ppm
Jun 21 07:47:48 dist-test-slave-lzx3 krb5kdc[8917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028068, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:48.866828 9292 init.cc:377] Logged in from keytab as kudu/127.0.192.195@KRBTEST.COM (short username kudu)
I20260621 07:47:48.867545 9292 webserver.cc:460] Webserver started at http://127.0.192.195:42877/ using document root <none> and password file <none>
I20260621 07:47:48.868027 9292 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:48.868110 9292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:48.868314 9292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:47:48.870258 9292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-2/data/instance:
uuid: "2b8dfb348d304f8993ae40747f0d4068"
format_stamp: "Formatted at 2026-06-21 07:47:48 on dist-test-slave-lzx3"
server_key: "3cb4d64abd09faea8da179f78fb545d2"
server_key_iv: "b7d247d2104f2f41703f8ff715f61610"
server_key_version: "encryptionkey@0"
I20260621 07:47:48.870795 9292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-2/wal/instance:
uuid: "2b8dfb348d304f8993ae40747f0d4068"
format_stamp: "Formatted at 2026-06-21 07:47:48 on dist-test-slave-lzx3"
server_key: "3cb4d64abd09faea8da179f78fb545d2"
server_key_iv: "b7d247d2104f2f41703f8ff715f61610"
server_key_version: "encryptionkey@0"
I20260621 07:47:48.874173 9292 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 07:47:48.876463 9307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:48.877697 9292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260621 07:47:48.877810 9292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-2/wal
uuid: "2b8dfb348d304f8993ae40747f0d4068"
format_stamp: "Formatted at 2026-06-21 07:47:48 on dist-test-slave-lzx3"
server_key: "3cb4d64abd09faea8da179f78fb545d2"
server_key_iv: "b7d247d2104f2f41703f8ff715f61610"
server_key_version: "encryptionkey@0"
I20260621 07:47:48.877916 9292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:48.899953 9292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:48.902696 9292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:48.902866 9292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:48.903337 9292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:47:48.904053 9292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:47:48.904119 9292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:48.904181 9292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:47:48.904219 9292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:48.912086 9292 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.195:35649
I20260621 07:47:48.912119 9420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.195:35649 every 8 connection(s)
I20260621 07:47:48.912884 9292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
I20260621 07:47:48.915648 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 9292
I20260621 07:47:48.915818 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-2/wal/instance
I20260621 07:47:48.916191 771 external_mini_cluster.cc:1428] Setting key 169efc609723d0c0a78b53dda59f6ff8
Jun 21 07:47:48 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028068, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:47:48.925765 9001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:48.914437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:42821 (local address 127.0.192.254:42251)
0621 07:47:48.914569 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:48.914572 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:48.915278 (+ 706us) server_negotiation.cc:407] Connection header received
0621 07:47:48.917088 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:48.917091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:48.917147 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:48.917248 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:48.918438 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:48.918925 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:48.920072 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:48.920307 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:48.920433 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:48.922564 (+ 2131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:48.922568 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:48.922572 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:48.922599 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:48.923772 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:48.924268 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:48.924273 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:48.924274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:48.924355 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:48.924767 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:48.924770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:48.924772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:48.924941 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:48.925042 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:48.925493 (+ 451us) server_negotiation.cc:299] Negotiation successful
0621 07:47:48.925601 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 07:47:48.926388 9423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:48.914735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.195:42821)
0621 07:47:48.915138 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:48.915177 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:48.916891 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:48.917398 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:48.917404 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:48.917806 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:48.918305 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:48.918317 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:48.919084 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:48.919088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:48.919941 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:48.919950 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:48.920463 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:48.920467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:48.920685 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:48.921114 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:48.921129 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:48.922407 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:47:48.923897 (+ 1490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:48.923903 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:48.923915 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:48.924165 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:48.924465 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:47:48.924468 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:47:48.924470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:47:48.924662 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:47:48.925052 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:47:48.925060 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:47:48.925269 (+ 209us) client_negotiation.cc:769] Sending connection context
0621 07:47:48.925436 (+ 167us) client_negotiation.cc:240] Negotiation successful
0621 07:47:48.925701 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":99,"threads_started":1}
I20260621 07:47:48.927286 9421 heartbeater.cc:347] Connected to a master server at 127.0.192.254:42251
I20260621 07:47:48.927462 9421 heartbeater.cc:464] Registering TS with master...
I20260621 07:47:48.927798 9421 heartbeater.cc:511] Master 127.0.192.254:42251 requested a full tablet report, sending...
I20260621 07:47:48.928651 8962 ts_manager.cc:194] Registered new tserver with Master: 2b8dfb348d304f8993ae40747f0d4068 (127.0.192.195:35649)
I20260621 07:47:48.929251 8962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.195@KRBTEST.COM'} at 127.0.192.195:42821
I20260621 07:47:48.929850 771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.192.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.192.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.192.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/oryx_127.0.192.193.keytab.
Entry for principal oryx/127.0.192.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/oryx_127.0.192.193.keytab.
WARNING: no policy specified for oryx/127.0.192.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.192.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.192.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/oryx_127.0.192.194.keytab.
Entry for principal oryx/127.0.192.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/oryx_127.0.192.194.keytab.
WARNING: no policy specified for oryx/127.0.192.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.192.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.192.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/oryx_127.0.192.195.keytab.
Entry for principal oryx/127.0.192.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/oryx_127.0.192.195.keytab.
I20260621 07:47:48.991377 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 9024
I20260621 07:47:48.997767 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 9158
I20260621 07:47:49.002719 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 9292
I20260621 07:47:49.007167 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 8930
I20260621 07:47:49.012496 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:42251
--webserver_interface=127.0.192.254
--webserver_port=46565
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:36829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:42251
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:47:49.109345 9430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:49.109558 9430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:49.109596 9430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:49.113183 9430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:47:49.113224 9430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:49.113241 9430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:49.113284 9430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:47:49.113313 9430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:47:49.118085 9430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:36829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:42251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:42251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=46565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:49.118882 9430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:49.119738 9430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:49.127127 9438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:49.127132 9436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:49.127158 9435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:49.127161 9430 server_base.cc:1034] running on GCE node
I20260621 07:47:49.127810 9430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:49.128782 9430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:49.129954 9430 hybrid_clock.cc:648] HybridClock initialized: now 1782028069129945 us; error 25 us; skew 500 ppm
Jun 21 07:47:49 dist-test-slave-lzx3 krb5kdc[8917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028069, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:49.132507 9430 init.cc:377] Logged in from keytab as kudu/127.0.192.254@KRBTEST.COM (short username kudu)
I20260621 07:47:49.133262 9430 webserver.cc:460] Webserver started at http://127.0.192.254:46565/ using document root <none> and password file <none>
I20260621 07:47:49.133744 9430 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:49.133842 9430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:49.137307 9430 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260621 07:47:49.139079 9445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:49.140178 9430 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:49.140307 9430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "dab035a0a5b146879905c833b365561e"
format_stamp: "Formatted at 2026-06-21 07:47:48 on dist-test-slave-lzx3"
server_key: "b8fedb78780bcae9f7c7da26d4bbe983"
server_key_iv: "75571f8492e208842abdef45e3ba5582"
server_key_version: "encryptionkey@0"
I20260621 07:47:49.140846 9430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:49.154656 9430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:49.157044 9430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:49.157215 9430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:49.163353 9430 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:42251
I20260621 07:47:49.163401 9497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:42251 every 8 connection(s)
I20260621 07:47:49.164216 9430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:47:49.166930 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 9430
I20260621 07:47:49.167376 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.193:36865
--local_ip_for_outbound_sockets=127.0.192.193
--tserver_master_addrs=127.0.192.254:42251
--webserver_port=43403
--webserver_interface=127.0.192.193
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:36829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.192.193
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/oryx_127.0.192.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:47:49.167865 9498 sys_catalog.cc:263] Verifying existing consensus state
I20260621 07:47:49.169778 9498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dab035a0a5b146879905c833b365561e: Bootstrap starting.
I20260621 07:47:49.184327 9498 log.cc:824] T 00000000000000000000000000000000 P dab035a0a5b146879905c833b365561e: Log is configured to *not* fsync() on all Append() calls
I20260621 07:47:49.189710 9498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dab035a0a5b146879905c833b365561e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 07:47:49.190037 9498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dab035a0a5b146879905c833b365561e: Bootstrap complete.
I20260621 07:47:49.193584 9498 raft_consensus.cc:348] T 00000000000000000000000000000000 P dab035a0a5b146879905c833b365561e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab035a0a5b146879905c833b365561e" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 42251 } }
I20260621 07:47:49.193887 9498 raft_consensus.cc:729] T 00000000000000000000000000000000 P dab035a0a5b146879905c833b365561e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dab035a0a5b146879905c833b365561e, State: Initialized, Role: FOLLOWER
I20260621 07:47:49.194351 9498 consensus_queue.cc:260] T 00000000000000000000000000000000 P dab035a0a5b146879905c833b365561e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab035a0a5b146879905c833b365561e" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 42251 } }
I20260621 07:47:49.194453 9498 raft_consensus.cc:388] T 00000000000000000000000000000000 P dab035a0a5b146879905c833b365561e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:47:49.194528 9498 raft_consensus.cc:482] T 00000000000000000000000000000000 P dab035a0a5b146879905c833b365561e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:47:49.194626 9498 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dab035a0a5b146879905c833b365561e [term 1 FOLLOWER]: Advancing to term 2
I20260621 07:47:49.196311 9498 raft_consensus.cc:504] T 00000000000000000000000000000000 P dab035a0a5b146879905c833b365561e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab035a0a5b146879905c833b365561e" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 42251 } }
I20260621 07:47:49.196663 9498 leader_election.cc:302] T 00000000000000000000000000000000 P dab035a0a5b146879905c833b365561e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dab035a0a5b146879905c833b365561e; no voters:
I20260621 07:47:49.196998 9498 leader_election.cc:288] T 00000000000000000000000000000000 P dab035a0a5b146879905c833b365561e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260621 07:47:49.197098 9502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dab035a0a5b146879905c833b365561e [term 2 FOLLOWER]: Leader election won for term 2
I20260621 07:47:49.197260 9502 raft_consensus.cc:686] T 00000000000000000000000000000000 P dab035a0a5b146879905c833b365561e [term 2 LEADER]: Becoming Leader. State: Replica: dab035a0a5b146879905c833b365561e, State: Running, Role: LEADER
I20260621 07:47:49.197543 9502 consensus_queue.cc:237] T 00000000000000000000000000000000 P dab035a0a5b146879905c833b365561e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab035a0a5b146879905c833b365561e" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 42251 } }
I20260621 07:47:49.198015 9498 sys_catalog.cc:564] T 00000000000000000000000000000000 P dab035a0a5b146879905c833b365561e [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:47:49.198436 9504 sys_catalog.cc:455] T 00000000000000000000000000000000 P dab035a0a5b146879905c833b365561e [sys.catalog]: SysCatalogTable state changed. Reason: New leader dab035a0a5b146879905c833b365561e. Latest consensus state: current_term: 2 leader_uuid: "dab035a0a5b146879905c833b365561e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab035a0a5b146879905c833b365561e" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 42251 } } }
I20260621 07:47:49.198541 9504 sys_catalog.cc:458] T 00000000000000000000000000000000 P dab035a0a5b146879905c833b365561e [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:49.198745 9503 sys_catalog.cc:455] T 00000000000000000000000000000000 P dab035a0a5b146879905c833b365561e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "dab035a0a5b146879905c833b365561e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab035a0a5b146879905c833b365561e" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 42251 } } }
I20260621 07:47:49.198851 9503 sys_catalog.cc:458] T 00000000000000000000000000000000 P dab035a0a5b146879905c833b365561e [sys.catalog]: This master's current role is: LEADER
I20260621 07:47:49.198956 9506 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:47:49.202018 9506 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:47:49.202739 9506 catalog_manager.cc:1259] Loaded cluster ID: 3b3386dc31d041d0a57b9bc607730f28
I20260621 07:47:49.202811 9506 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:47:49.204901 9506 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:47:49.205498 9506 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dab035a0a5b146879905c833b365561e: Loaded TSK: 0
I20260621 07:47:49.206637 9506 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 07:47:49.270561 9500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:49.270843 9500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:49.270931 9500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:49.275849 9500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:49.275974 9500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:49.276152 9500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.193
I20260621 07:47:49.281373 9500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:36829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/oryx_127.0.192.193.keytab
--principal=oryx/127.0.192.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.193:36865
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.192.193
--webserver_port=43403
--tserver_master_addrs=127.0.192.254:42251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.193
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:49.282231 9500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:49.283087 9500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:49.292296 9524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:49.292430 9500 server_base.cc:1034] running on GCE node
W20260621 07:47:49.292501 9525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:49.292346 9527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:49.292939 9500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:49.293448 9500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:49.294626 9500 hybrid_clock.cc:648] HybridClock initialized: now 1782028069294605 us; error 36 us; skew 500 ppm
Jun 21 07:47:49 dist-test-slave-lzx3 krb5kdc[8917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028069, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.192.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:49.297202 9500 init.cc:377] Logged in from keytab as oryx/127.0.192.193@KRBTEST.COM (short username oryx)
I20260621 07:47:49.297911 9500 webserver.cc:460] Webserver started at http://127.0.192.193:43403/ using document root <none> and password file <none>
I20260621 07:47:49.298449 9500 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:49.298511 9500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:49.302088 9500 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260621 07:47:49.303845 9534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:49.305009 9500 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260621 07:47:49.305130 9500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-0/wal
uuid: "9f3cd4138e8d4a1cbc9d01b7dd5c13a8"
format_stamp: "Formatted at 2026-06-21 07:47:48 on dist-test-slave-lzx3"
server_key: "9bdbec867d62b8d86e89d075007f165d"
server_key_iv: "a2e5be60ba03842c1ffd4fc2fda5ebb5"
server_key_version: "encryptionkey@0"
I20260621 07:47:49.305581 9500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:49.322247 9500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:49.324934 9500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:49.325109 9500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:49.325644 9500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:47:49.326390 9500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:47:49.326462 9500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:49.326524 9500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:47:49.326581 9500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:49.334254 9500 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.193:36865
I20260621 07:47:49.334323 9647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.193:36865 every 8 connection(s)
I20260621 07:47:49.335090 9500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
I20260621 07:47:49.343278 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 9500
I20260621 07:47:49.343654 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.194:36135
--local_ip_for_outbound_sockets=127.0.192.194
--tserver_master_addrs=127.0.192.254:42251
--webserver_port=43723
--webserver_interface=127.0.192.194
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:36829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.192.194
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/oryx_127.0.192.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 07:47:49 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.193@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:47:49 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.193@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:47:49.346606 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:49.337088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.193:45671)
0621 07:47:49.337525 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:49.337559 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:49.338616 (+ 1057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:49.339607 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:49.339615 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:49.339987 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:49.340392 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:49.340402 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:49.341540 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:49.341543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:49.342317 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:49.342326 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:49.342950 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:49.342953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:49.343115 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:49.343577 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:49.343590 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:49.345808 (+ 2218us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":109,"threads_started":1}
W20260621 07:47:49.346597 9650 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:49.336972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:45671 (local address 127.0.192.254:42251)
0621 07:47:49.337387 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:49.337404 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:49.337658 (+ 254us) server_negotiation.cc:407] Connection header received
0621 07:47:49.338946 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:49.338954 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:49.339222 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:49.339525 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:49.340589 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:49.341413 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:49.342431 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:49.342813 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:49.342961 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:49.345975 (+ 3014us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.193:45671: BlockingRecv error: recv got EOF from 127.0.192.193:45671 (error 108)
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":98,"threads_started":1}
W20260621 07:47:49.347040 9648 heartbeater.cc:644] Failed to heartbeat to 127.0.192.254:42251 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.192.254:42251: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Jun 21 07:47:49 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.193@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:47:49 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.193@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
W20260621 07:47:49.352002 9650 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:49.347490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:34583 (local address 127.0.192.254:42251)
0621 07:47:49.347730 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:49.347733 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:49.347741 (+ 8us) server_negotiation.cc:407] Connection header received
0621 07:47:49.347764 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:49.347766 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:49.347794 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:49.347874 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:49.348191 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:49.348665 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:49.349380 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:49.349596 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:49.349700 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:49.351927 (+ 2227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.193:34583: BlockingRecv error: recv got EOF from 127.0.192.193:34583 (error 108)
Metrics: {"server-negotiator.queue_time_us":192}
I20260621 07:47:49.352020 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:49.347433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.193:34583)
0621 07:47:49.347557 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:49.347570 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:49.347635 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:49.347885 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:49.347888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:49.348021 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:49.348091 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:49.348095 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:49.348755 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:49.348760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:49.349266 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:49.349274 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:49.349700 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:49.349703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:49.349799 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:49.350216 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:49.350227 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:49.351841 (+ 1614us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
Jun 21 07:47:49 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.193@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:47:49 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.193@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:47:49.356877 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:49.352455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.193:54535)
0621 07:47:49.352579 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:49.352591 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:49.352658 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:49.352877 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:49.352880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:49.353015 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:49.353070 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:49.353074 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:49.353683 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:49.353687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:49.354166 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:49.354172 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:49.354559 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:49.354562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:49.354631 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:49.355056 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:49.355065 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:49.356739 (+ 1674us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260621 07:47:49.356961 9650 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:49.352512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:54535 (local address 127.0.192.254:42251)
0621 07:47:49.352617 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:49.352622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:49.352635 (+ 13us) server_negotiation.cc:407] Connection header received
0621 07:47:49.352756 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:49.352758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:49.352796 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:49.352867 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:49.353167 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:49.353588 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:49.354270 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:49.354457 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:49.354560 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:49.356843 (+ 2283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.193:54535: BlockingRecv error: recv got EOF from 127.0.192.193:54535 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260621 07:47:49.357133 9648 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 07:47:49.443823 9652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:49.444039 9652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:49.444108 9652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:49.447527 9652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:49.447582 9652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:49.447688 9652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.194
I20260621 07:47:49.452334 9652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:36829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/oryx_127.0.192.194.keytab
--principal=oryx/127.0.192.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.194:36135
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.192.194
--webserver_port=43723
--tserver_master_addrs=127.0.192.254:42251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.194
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:49.453173 9652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:49.454026 9652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:49.462445 9660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:49.462558 9657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:49.462528 9658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:49.462625 9652 server_base.cc:1034] running on GCE node
I20260621 07:47:49.463011 9652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:49.463433 9652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:49.464600 9652 hybrid_clock.cc:648] HybridClock initialized: now 1782028069464592 us; error 23 us; skew 500 ppm
Jun 21 07:47:49 dist-test-slave-lzx3 krb5kdc[8917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028069, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.192.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:49.466761 9652 init.cc:377] Logged in from keytab as oryx/127.0.192.194@KRBTEST.COM (short username oryx)
I20260621 07:47:49.467473 9652 webserver.cc:460] Webserver started at http://127.0.192.194:43723/ using document root <none> and password file <none>
I20260621 07:47:49.467872 9652 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:49.467928 9652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:49.471071 9652 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:49.472684 9667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:49.473637 9652 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:49.473713 9652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-1/wal
uuid: "10877c6410584af4ad1167ec386c70e4"
format_stamp: "Formatted at 2026-06-21 07:47:48 on dist-test-slave-lzx3"
server_key: "c454da0f4b14279cd4b877801e84ec54"
server_key_iv: "51fde3e2c6a40f5c87a0802d55af36a3"
server_key_version: "encryptionkey@0"
I20260621 07:47:49.474092 9652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:49.483374 9652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:49.486001 9652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:49.486119 9652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:49.486522 9652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:47:49.487128 9652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:47:49.487167 9652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:49.487197 9652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:47:49.487216 9652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:49.494938 9652 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.194:36135
I20260621 07:47:49.494961 9780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.194:36135 every 8 connection(s)
I20260621 07:47:49.495690 9652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
I20260621 07:47:49.498924 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 9652
I20260621 07:47:49.499408 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.195:35649
--local_ip_for_outbound_sockets=127.0.192.195
--tserver_master_addrs=127.0.192.254:42251
--webserver_port=42877
--webserver_interface=127.0.192.195
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:36829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.192.195
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/oryx_127.0.192.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 07:47:49 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.194@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:47:49 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.194@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
W20260621 07:47:49.506085 9650 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:49.497272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:37233 (local address 127.0.192.254:42251)
0621 07:47:49.497393 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:49.497397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:49.498090 (+ 693us) server_negotiation.cc:407] Connection header received
0621 07:47:49.499681 (+ 1591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:49.499686 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:49.499751 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:49.499811 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:49.501353 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:49.501904 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:49.502882 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:49.503078 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:49.503179 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:49.505968 (+ 2789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.194:37233: BlockingRecv error: recv got EOF from 127.0.192.194:37233 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260621 07:47:49.506551 9783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:49.497553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.194:37233)
0621 07:47:49.497947 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:49.497991 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:49.499429 (+ 1438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:49.500193 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:49.500203 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:49.500737 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:49.501220 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:49.501231 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:49.502043 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:49.502045 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:49.502762 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:49.502770 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:49.503209 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:49.503213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:49.503356 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:49.503805 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:49.503818 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:49.505839 (+ 2021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":94,"threads_started":1}
W20260621 07:47:49.506897 9781 heartbeater.cc:644] Failed to heartbeat to 127.0.192.254:42251 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.192.254:42251: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Jun 21 07:47:49 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.194@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:47:49 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.194@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:47:49.512121 9783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:49.507281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.194:40099)
0621 07:47:49.507437 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:49.507451 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:49.507533 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:49.507773 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:49.507778 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:49.507929 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:49.508054 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:49.508064 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:49.508820 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:49.508823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:49.509657 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:49.509667 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:49.510058 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:49.510062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:49.510154 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:49.510527 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:49.510538 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:49.512025 (+ 1487us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91}
W20260621 07:47:49.512193 9650 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:49.507341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:40099 (local address 127.0.192.254:42251)
0621 07:47:49.507442 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:49.507444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:49.507491 (+ 47us) server_negotiation.cc:407] Connection header received
0621 07:47:49.507613 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:49.507614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:49.507661 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:49.507733 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:49.508187 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:49.508697 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:49.509761 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:49.509956 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:49.510034 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:49.512124 (+ 2090us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.194:40099: BlockingRecv error: recv got EOF from 127.0.192.194:40099 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 21 07:47:49 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.194@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:47:49 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.194@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:47:49.516928 9783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:49.512713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.194:53857)
0621 07:47:49.512786 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:49.512798 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:49.512857 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:49.513068 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:49.513071 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:49.513198 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:49.513251 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:49.513255 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:49.513877 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:49.513880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:49.514359 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:49.514365 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:49.514757 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:49.514760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:49.514823 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:49.515193 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:49.515202 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:49.516794 (+ 1592us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":22}
W20260621 07:47:49.516997 9650 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:49.512782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:53857 (local address 127.0.192.254:42251)
0621 07:47:49.512868 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:49.512873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:49.512886 (+ 13us) server_negotiation.cc:407] Connection header received
0621 07:47:49.512918 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:49.512920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:49.512949 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:49.513057 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:49.513350 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:49.513774 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:49.514465 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:49.514667 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:49.514757 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:49.516898 (+ 2141us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.194:53857: BlockingRecv error: recv got EOF from 127.0.192.194:53857 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260621 07:47:49.517074 9781 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 07:47:49.597996 9784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:47:49.598201 9784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:47:49.598268 9784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:47:49.602128 9784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:47:49.602174 9784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:47:49.602260 9784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.195
I20260621 07:47:49.606899 9784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:36829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/krb5kdc/oryx_127.0.192.195.keytab
--principal=oryx/127.0.192.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.195:35649
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.192.195
--webserver_port=42877
--tserver_master_addrs=127.0.192.254:42251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.195
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:47:49.607818 9784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:47:49.608778 9784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:47:49.617411 9792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:49.617394 9790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:47:49.617398 9789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:47:49.617678 9784 server_base.cc:1034] running on GCE node
I20260621 07:47:49.618046 9784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:47:49.618582 9784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:47:49.619769 9784 hybrid_clock.cc:648] HybridClock initialized: now 1782028069619754 us; error 28 us; skew 500 ppm
Jun 21 07:47:49 dist-test-slave-lzx3 krb5kdc[8917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028069, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.192.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:47:49.622293 9784 init.cc:377] Logged in from keytab as oryx/127.0.192.195@KRBTEST.COM (short username oryx)
I20260621 07:47:49.623015 9784 webserver.cc:460] Webserver started at http://127.0.192.195:42877/ using document root <none> and password file <none>
I20260621 07:47:49.623521 9784 fs_manager.cc:359] Metadata directory not provided
I20260621 07:47:49.623612 9784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:47:49.626921 9784 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:47:49.628793 9799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:49.630353 9784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 07:47:49.630517 9784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-2/wal
uuid: "2b8dfb348d304f8993ae40747f0d4068"
format_stamp: "Formatted at 2026-06-21 07:47:48 on dist-test-slave-lzx3"
server_key: "3cb4d64abd09faea8da179f78fb545d2"
server_key_iv: "b7d247d2104f2f41703f8ff715f61610"
server_key_version: "encryptionkey@0"
I20260621 07:47:49.631222 9784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:47:49.648079 9784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:47:49.650854 9784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:47:49.651028 9784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:47:49.651537 9784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:47:49.652267 9784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:47:49.652309 9784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:49.652374 9784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:47:49.652405 9784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:47:49.661988 9784 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.195:35649
I20260621 07:47:49.662052 9912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.195:35649 every 8 connection(s)
I20260621 07:47:49.662889 9784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
I20260621 07:47:49.664958 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 9784
I20260621 07:47:49.676877 9917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:49.665867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51750 (local address 127.0.192.254:42251)
0621 07:47:49.666080 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:49.666096 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:49.666113 (+ 17us) server_negotiation.cc:407] Connection header received
0621 07:47:49.666162 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:49.666167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:49.666223 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:49.666324 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:49.667176 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:49.668063 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:49.669537 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:49.669982 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:49.670104 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:49.671522 (+ 1418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:47:49.671530 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:47:49.671549 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:47:49.671575 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:47:49.673977 (+ 2402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:49.674441 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:49.674448 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:49.674457 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:49.674629 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:47:49.674944 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:47:49.674950 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:47:49.674953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:47:49.675392 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:47:49.675591 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0621 07:47:49.676001 (+ 410us) server_negotiation.cc:299] Negotiation successful
0621 07:47:49.676357 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":58,"threads_started":1}
Jun 21 07:47:49 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.195@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:47:49 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.195@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
W20260621 07:47:49.680009 9650 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:49.664525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:45441 (local address 127.0.192.254:42251)
0621 07:47:49.664683 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:49.664688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:49.665424 (+ 736us) server_negotiation.cc:407] Connection header received
0621 07:47:49.667284 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:49.667287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:49.667331 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:49.667398 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:49.668473 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:49.669388 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:49.670882 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:49.671243 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:49.671353 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:49.679860 (+ 8507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.195:45441: BlockingRecv error: recv got EOF from 127.0.192.195:45441 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 07:47:49.680274 9915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:49.664890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.195:45441)
0621 07:47:49.665300 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:49.665343 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:49.667099 (+ 1756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:49.667544 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:49.667549 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:49.667874 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:49.668340 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:49.668357 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:49.669542 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:49.669548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:49.670760 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:49.670770 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:49.671393 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:49.671397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:49.671547 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:49.676368 (+ 4821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:49.676386 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:49.679467 (+ 3081us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":130,"threads_started":1}
W20260621 07:47:49.680743 9913 heartbeater.cc:644] Failed to heartbeat to 127.0.192.254:42251 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.192.254:42251: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Jun 21 07:47:49 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.195@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:47:49 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.195@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:47:49.685527 9915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:49.681177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.195:49453)
0621 07:47:49.681278 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:49.681292 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:49.681368 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:49.681588 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:49.681590 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:49.681739 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:49.681814 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:49.681819 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:49.682490 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:49.682496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:49.683044 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:49.683052 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:49.683438 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:49.683441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:49.683518 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:49.683898 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:49.683911 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:49.685411 (+ 1500us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260621 07:47:49.685604 9917 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:49.681229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:49453 (local address 127.0.192.254:42251)
0621 07:47:49.681333 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:49.681337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:49.681350 (+ 13us) server_negotiation.cc:407] Connection header received
0621 07:47:49.681443 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:49.681447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:49.681481 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:49.681577 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:49.681923 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:49.682389 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:49.683146 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:49.683348 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:49.683439 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:49.685516 (+ 2077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.195:49453: BlockingRecv error: recv got EOF from 127.0.192.195:49453 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 21 07:47:49 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.195@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:47:49 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.195@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:47:49.690083 9915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:49.685962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.195:56545)
0621 07:47:49.686043 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:49.686055 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:49.686118 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:49.686339 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:49.686341 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:49.686469 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:49.686522 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:49.686526 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:49.687092 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:49.687097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:49.687528 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:49.687534 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:49.687900 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:49.687903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:49.687968 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:49.688424 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:49.688433 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:49.689956 (+ 1523us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":21}
W20260621 07:47:49.690255 9913 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 07:47:49.690140 9917 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:49.686037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:56545 (local address 127.0.192.254:42251)
0621 07:47:49.686185 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:49.686190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:49.686199 (+ 9us) server_negotiation.cc:407] Connection header received
0621 07:47:49.686224 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:49.686227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:49.686259 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:49.686329 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:49.686614 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:49.687004 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:49.687628 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:49.687812 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:49.687899 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:49.690051 (+ 2152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.195:56545: BlockingRecv error: recv got EOF from 127.0.192.195:56545 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 21 07:47:50 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.193@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:47:50 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.193@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:47:50.362540 9918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:50.357656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.193:55753)
0621 07:47:50.357880 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:50.357902 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:50.357990 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:50.358279 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:50.358282 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:50.358495 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:50.358629 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:50.358642 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:50.359270 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:50.359272 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:50.359825 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:50.359832 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:50.360230 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:50.360233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:50.360297 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:50.360844 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:50.360856 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:50.362436 (+ 1580us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":73,"threads_started":1}
W20260621 07:47:50.362654 9919 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:50.357690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:55753 (local address 127.0.192.254:42251)
0621 07:47:50.357931 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:50.357936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:50.357949 (+ 13us) server_negotiation.cc:407] Connection header received
0621 07:47:50.358083 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:50.358085 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:50.358144 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:50.358233 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:50.358740 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:50.359192 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:50.359921 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:50.360140 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:50.360215 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:50.362541 (+ 2326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.193:55753: BlockingRecv error: recv got EOF from 127.0.192.193:55753 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":91,"threads_started":1}
Jun 21 07:47:50 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.194@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:47:50 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.194@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:47:50.522171 9920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:50.517551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.194:46195)
0621 07:47:50.517768 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:50.517779 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:50.517848 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:50.518030 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:50.518031 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:50.518178 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:50.518275 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:50.518279 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:50.518918 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:50.518920 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:50.519451 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:50.519457 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:50.519843 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:50.519845 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:50.519914 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:50.520432 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:50.520443 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:50.522080 (+ 1637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":80,"threads_started":1}
W20260621 07:47:50.522293 9919 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:50.517598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:46195 (local address 127.0.192.254:42251)
0621 07:47:50.517813 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:50.517816 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:50.517827 (+ 11us) server_negotiation.cc:407] Connection header received
0621 07:47:50.517922 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:50.517924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:50.517961 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:50.518020 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:50.518356 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:50.518837 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:50.519546 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:50.519754 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:50.519827 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:50.522187 (+ 2360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.194:46195: BlockingRecv error: recv got EOF from 127.0.192.194:46195 (error 108)
Metrics: {"server-negotiator.queue_time_us":143}
Jun 21 07:47:50 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.195@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:47:50 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.195@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:47:50.695839 9921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:50.690935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.195:43499)
0621 07:47:50.691182 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:50.691192 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:50.691303 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:50.691534 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:50.691537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:50.691704 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:50.691783 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:50.691787 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:50.692432 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:50.692436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:50.692944 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:50.692952 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:50.693477 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:50.693480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:50.693557 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:50.694005 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:50.694015 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:50.695681 (+ 1666us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":72,"threads_started":1}
W20260621 07:47:50.695906 9919 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:50.690991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:43499 (local address 127.0.192.254:42251)
0621 07:47:50.691136 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:50.691140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:50.691255 (+ 115us) server_negotiation.cc:407] Connection header received
0621 07:47:50.691387 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:50.691390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:50.691436 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:50.691508 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:50.691886 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:50.692337 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:50.693052 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:50.693311 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:50.693415 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:50.695778 (+ 2363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.195:43499: BlockingRecv error: recv got EOF from 127.0.192.195:43499 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 21 07:47:51 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.193@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:47:51 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.193@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:47:51.367929 9922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:51.363257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.193:36005)
0621 07:47:51.363492 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:51.363504 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:51.363582 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:51.363777 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:51.363778 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:51.363924 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:51.364006 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:51.364010 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:51.364626 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:51.364628 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:51.365121 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:51.365127 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:51.365495 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:51.365497 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:51.365549 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:51.366066 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:51.366079 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:51.367771 (+ 1692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":93,"threads_started":1}
W20260621 07:47:51.368084 9923 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:51.363309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:36005 (local address 127.0.192.254:42251)
0621 07:47:51.363524 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:51.363527 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:51.363543 (+ 16us) server_negotiation.cc:407] Connection header received
0621 07:47:51.363661 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:51.363663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:51.363702 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:51.363768 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:51.364088 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:51.364543 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:51.365222 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:51.365413 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:51.365480 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:51.367929 (+ 2449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.193:36005: BlockingRecv error: recv got EOF from 127.0.192.193:36005 (error 108)
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":53,"threads_started":1}
Jun 21 07:47:51 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.194@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:47:51 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.194@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:47:51.527855 9924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:51.522889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.194:37051)
0621 07:47:51.523164 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:51.523178 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:51.523278 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:51.523515 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:51.523518 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:51.523700 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:51.523779 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:51.523783 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:51.524439 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:51.524444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:51.525024 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:51.525032 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:51.525420 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:51.525423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:51.525493 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:51.526026 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:51.526039 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:51.527763 (+ 1724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":82,"threads_started":1}
W20260621 07:47:51.527988 9923 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:51.523039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:37051 (local address 127.0.192.254:42251)
0621 07:47:51.523166 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:51.523171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:51.523231 (+ 60us) server_negotiation.cc:407] Connection header received
0621 07:47:51.523378 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:51.523384 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:51.523423 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:51.523496 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:51.523881 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:51.524338 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:51.525126 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:51.525324 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:51.525419 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:51.527873 (+ 2454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.194:37051: BlockingRecv error: recv got EOF from 127.0.192.194:37051 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 21 07:47:51 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.195@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:47:51 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.195@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:47:51.701606 9925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:51.696517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.195:45943)
0621 07:47:51.696788 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:51.696803 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:51.696918 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:51.697173 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:51.697175 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:51.697345 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:51.697427 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:51.697432 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:51.698156 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:51.698159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:51.698788 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:51.698796 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:51.699162 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:51.699164 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:51.699229 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:51.699781 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:51.699792 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:51.701489 (+ 1697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":126,"threads_started":1}
W20260621 07:47:51.701669 9923 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:51.696670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:45943 (local address 127.0.192.254:42251)
0621 07:47:51.696788 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:51.696792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:51.696878 (+ 86us) server_negotiation.cc:407] Connection header received
0621 07:47:51.697023 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:51.697026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:51.697070 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:51.697127 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:51.697536 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:51.698068 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:51.698903 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:51.699082 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:51.699151 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:51.701592 (+ 2441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.195:45943: BlockingRecv error: recv got EOF from 127.0.192.195:45943 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 21 07:47:52 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.193@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:47:52 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.193@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:47:52.373602 9926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:52.368812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.193:43981)
0621 07:47:52.369081 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:52.369099 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:52.369183 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:52.369459 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:52.369462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:52.369636 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:52.369731 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:52.369736 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:52.370435 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:52.370437 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:52.370984 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:52.370990 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:52.371333 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:52.371335 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:52.371387 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:52.371871 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:52.371882 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:52.373507 (+ 1625us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":99,"threads_started":1}
W20260621 07:47:52.373734 9927 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:52.368967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:43981 (local address 127.0.192.254:42251)
0621 07:47:52.369210 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:52.369213 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:52.369226 (+ 13us) server_negotiation.cc:407] Connection header received
0621 07:47:52.369258 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:52.369261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:52.369299 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:52.369376 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:52.369846 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:52.370347 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:52.371070 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:52.371254 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:52.371319 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:52.373618 (+ 2299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.193:43981: BlockingRecv error: recv got EOF from 127.0.192.193:43981 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":95,"threads_started":1}
Jun 21 07:47:52 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.194@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:47:52 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.194@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:47:52.533794 9928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:52.528476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.194:57323)
0621 07:47:52.528707 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:52.528719 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:52.528808 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:52.529009 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:52.529011 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:52.529173 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:52.529264 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:52.529269 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:52.530433 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:52.530436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:52.530978 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:52.530985 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:52.531402 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:52.531404 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:52.531459 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:52.531946 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:52.531957 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:52.533646 (+ 1689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":94,"threads_started":1}
W20260621 07:47:52.533828 9927 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:52.528614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:57323 (local address 127.0.192.254:42251)
0621 07:47:52.528735 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:52.528738 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:52.528764 (+ 26us) server_negotiation.cc:407] Connection header received
0621 07:47:52.528877 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:52.528879 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:52.528918 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:52.528978 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:52.529483 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:52.530314 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:52.531102 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:52.531313 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:52.531386 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:52.533746 (+ 2360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.194:57323: BlockingRecv error: recv got EOF from 127.0.192.194:57323 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 21 07:47:52 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.195@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:47:52 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.195@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:47:52.707340 9929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:52.702377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.195:49921)
0621 07:47:52.702652 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:52.702670 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:52.702784 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:52.703029 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:52.703032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:52.703236 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:52.703320 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:52.703327 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:52.704002 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:52.704006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:52.704496 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:52.704503 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:52.704918 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:52.704921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:52.704994 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:52.705502 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:52.705517 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:52.707189 (+ 1672us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":94,"threads_started":1}
W20260621 07:47:52.707397 9927 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:52.702461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:49921 (local address 127.0.192.254:42251)
0621 07:47:52.702603 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:52.702607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:52.702738 (+ 131us) server_negotiation.cc:407] Connection header received
0621 07:47:52.702884 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:52.702886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:52.702932 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:52.703012 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:52.703430 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:52.703905 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:52.704628 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:52.704820 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:52.704918 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:52.707292 (+ 2374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.195:49921: BlockingRecv error: recv got EOF from 127.0.192.195:49921 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 21 07:47:53 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.193@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:47:53 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.193@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:47:53.384032 9930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:53.374310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.193:55717)
0621 07:47:53.374613 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:53.374626 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:53.374724 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:53.374951 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:53.374953 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:53.375116 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:53.375206 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:53.375211 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:53.375838 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:53.375840 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:53.376376 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:53.376385 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:53.376933 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:53.376936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:53.376997 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:53.382170 (+ 5173us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:53.382181 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:53.383868 (+ 1687us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":133,"threads_started":1}
W20260621 07:47:53.384128 9931 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:53.374424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:55717 (local address 127.0.192.254:42251)
0621 07:47:53.374637 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:53.374647 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:53.374677 (+ 30us) server_negotiation.cc:407] Connection header received
0621 07:47:53.374803 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:53.374805 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:53.374843 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:53.374920 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:53.375308 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:53.375755 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:53.376537 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:53.376825 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:53.376921 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:53.384000 (+ 7079us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.193:55717: BlockingRecv error: recv got EOF from 127.0.192.193:55717 (error 108)
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":60,"threads_started":1}
Jun 21 07:47:53 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.194@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:47:53 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.194@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:47:53.539438 9932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:53.534630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.194:37455)
0621 07:47:53.534857 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:53.534869 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:53.534968 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:53.535174 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:53.535175 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:53.535325 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:53.535415 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:53.535420 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:53.536042 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:53.536044 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:53.536703 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:53.536712 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:53.537127 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:53.537129 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:53.537186 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:53.537616 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:53.537625 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:53.539284 (+ 1659us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":86,"threads_started":1}
W20260621 07:47:53.539546 9931 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:53.534661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:37455 (local address 127.0.192.254:42251)
0621 07:47:53.534788 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:53.534792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:53.534944 (+ 152us) server_negotiation.cc:407] Connection header received
0621 07:47:53.535037 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:53.535039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:53.535078 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:53.535139 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:53.535499 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:53.535956 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:53.536831 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:53.537039 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:53.537114 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:53.539425 (+ 2311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.194:37455: BlockingRecv error: recv got EOF from 127.0.192.194:37455 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 21 07:47:53 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.195@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:47:53 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.195@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:47:53.713716 9933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:53.708240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.195:51035)
0621 07:47:53.708518 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:53.708531 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:53.708688 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:53.708947 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:53.708950 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:53.709196 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:53.709285 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:53.709291 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:53.709964 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:53.709970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:53.710592 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:53.710602 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:53.711178 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:53.711182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:53.711296 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:53.711806 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:53.711821 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:53.713557 (+ 1736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":87,"threads_started":1}
W20260621 07:47:53.713771 9931 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:53.708295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:51035 (local address 127.0.192.254:42251)
0621 07:47:53.708450 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:53.708456 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:53.708630 (+ 174us) server_negotiation.cc:407] Connection header received
0621 07:47:53.708779 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:53.708782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:53.708832 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:53.708930 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:53.709394 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:53.709854 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:53.710719 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:53.711055 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:53.711179 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:53.713669 (+ 2490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.195:51035: BlockingRecv error: recv got EOF from 127.0.192.195:51035 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 21 07:47:54 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.193@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:47:54 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.193@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:47:54.389848 9934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:54.384735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.193:34229)
0621 07:47:54.385056 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:54.385068 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:54.385153 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:54.385354 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:54.385356 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:54.385501 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:54.385589 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:54.385594 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:54.386452 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:54.386454 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:54.387083 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:54.387092 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:54.387489 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:54.387492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:54.387552 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:54.388003 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:54.388013 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:54.389706 (+ 1693us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":71,"threads_started":1}
W20260621 07:47:54.389909 9935 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:54.384849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:34229 (local address 127.0.192.254:42251)
0621 07:47:54.385054 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:54.385057 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:54.385131 (+ 74us) server_negotiation.cc:407] Connection header received
0621 07:47:54.385219 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:54.385221 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:54.385257 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:54.385324 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:54.385694 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:54.386322 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:54.387193 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:54.387388 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:54.387486 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:54.389805 (+ 2319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.193:34229: BlockingRecv error: recv got EOF from 127.0.192.193:34229 (error 108)
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":75,"threads_started":1}
Jun 21 07:47:54 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.194@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:47:54 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.194@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:47:54.545344 9936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:54.540149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.194:57293)
0621 07:47:54.540457 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:54.540473 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:54.540606 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:54.540902 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:54.540905 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:54.541103 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:54.541233 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:54.541240 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:54.541925 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:54.541928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:54.542424 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:54.542432 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:54.542789 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:54.542792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:54.542846 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:54.543350 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:54.543360 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:54.545252 (+ 1892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":125,"threads_started":1}
W20260621 07:47:54.545459 9935 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:54.540319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:57293 (local address 127.0.192.254:42251)
0621 07:47:54.540458 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:54.540464 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:54.540545 (+ 81us) server_negotiation.cc:407] Connection header received
0621 07:47:54.540712 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:54.540715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:54.540763 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:54.540852 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:54.541356 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:54.541830 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:54.542512 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:54.542706 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:54.542775 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:54.545359 (+ 2584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.194:57293: BlockingRecv error: recv got EOF from 127.0.192.194:57293 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 21 07:47:54 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.195@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:47:54 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.195@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:47:54.719210 9937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:54.714350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.195:59711)
0621 07:47:54.714578 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:54.714590 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:54.714677 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:54.714876 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:54.714878 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:54.715041 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:54.715124 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:54.715129 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:54.715876 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:54.715879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:54.716460 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:54.716470 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:54.716882 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:54.716884 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:54.716951 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:54.717432 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:54.717444 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:54.719104 (+ 1660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":60,"threads_started":1}
W20260621 07:47:54.719368 9935 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:54.714456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:59711 (local address 127.0.192.254:42251)
0621 07:47:54.714604 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:54.714607 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:54.714637 (+ 30us) server_negotiation.cc:407] Connection header received
0621 07:47:54.714748 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:54.714749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:54.714788 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:54.714846 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:54.715208 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:54.715662 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:54.716588 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:54.716787 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:54.716858 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:54.719224 (+ 2366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.195:59711: BlockingRecv error: recv got EOF from 127.0.192.195:59711 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 21 07:47:55 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.193@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:47:55 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.193@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:47:55.395233 9938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:55.390654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.193:33827)
0621 07:47:55.390916 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:55.390929 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:55.391015 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:55.391249 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:55.391251 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:55.391419 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:55.391510 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:55.391514 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:55.392180 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:55.392182 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:55.392678 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:55.392684 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:55.393030 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:55.393032 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:55.393081 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:55.393514 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:55.393523 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:55.395136 (+ 1613us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":75,"threads_started":1}
W20260621 07:47:55.395437 9939 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:55.390748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:33827 (local address 127.0.192.254:42251)
0621 07:47:55.391021 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:55.391025 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:55.391039 (+ 14us) server_negotiation.cc:407] Connection header received
0621 07:47:55.391078 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:55.391080 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:55.391134 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:55.391218 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:55.391633 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:55.392094 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:55.392759 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:55.392949 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:55.393021 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:55.395305 (+ 2284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.193:33827: BlockingRecv error: recv got EOF from 127.0.192.193:33827 (error 108)
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":77,"threads_started":1}
Jun 21 07:47:55 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.194@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:47:55 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.194@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:47:55.551091 9940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:55.546132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.194:60361)
0621 07:47:55.546373 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:55.546386 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:55.546474 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:55.546692 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:55.546694 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:55.546872 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:55.546954 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:55.546958 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:55.547582 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:55.547584 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:55.548072 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:55.548078 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:55.548474 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:55.548495 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:55.548690 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:55.549147 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:55.549157 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:55.550930 (+ 1773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":89,"threads_started":1}
W20260621 07:47:55.551146 9939 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:55.546212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:60361 (local address 127.0.192.254:42251)
0621 07:47:55.546336 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:55.546339 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:55.546446 (+ 107us) server_negotiation.cc:407] Connection header received
0621 07:47:55.546544 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:55.546546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:55.546585 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:55.546658 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:55.547038 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:55.547502 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:55.548162 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:55.548359 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:55.548434 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:55.551038 (+ 2604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.194:60361: BlockingRecv error: recv got EOF from 127.0.192.194:60361 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 21 07:47:55 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.195@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:47:55 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.195@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:47:55.725075 9941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:55.719973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.195:59235)
0621 07:47:55.720210 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:55.720221 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:55.720335 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:55.720672 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:55.720674 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:55.720904 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:55.721034 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:55.721043 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:55.721884 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:55.721886 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:55.722411 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:55.722419 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:55.722766 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:55.722768 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:55.722820 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:55.723234 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:55.723244 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:55.724916 (+ 1672us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":85,"threads_started":1}
W20260621 07:47:55.725127 9939 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:55.720014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:59235 (local address 127.0.192.254:42251)
0621 07:47:55.720137 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:55.720142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:55.720294 (+ 152us) server_negotiation.cc:407] Connection header received
0621 07:47:55.720418 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:55.720420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:55.720522 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:55.720631 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:55.721164 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:55.721788 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:55.722498 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:55.722684 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:55.722751 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:55.725022 (+ 2271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.195:59235: BlockingRecv error: recv got EOF from 127.0.192.195:59235 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 21 07:47:56 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.193@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:47:56 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.193@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:47:56.401605 9943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:56.395952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.193:34461)
0621 07:47:56.396304 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:56.396323 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:56.396492 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:56.396808 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:56.396810 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:56.396975 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:56.397067 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:56.397071 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:56.397825 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:56.397827 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:56.398346 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:56.398353 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:56.399043 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:56.399047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:56.399146 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:56.399738 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:56.399762 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:56.401499 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":83,"threads_started":1}
W20260621 07:47:56.401773 9942 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:56.396014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:34461 (local address 127.0.192.254:42251)
0621 07:47:56.396238 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:56.396243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:56.396585 (+ 342us) server_negotiation.cc:407] Connection header received
0621 07:47:56.396625 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:56.396629 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:56.396684 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:56.396787 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:56.397161 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:56.397747 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:56.398511 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:56.398905 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:56.399039 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:56.401624 (+ 2585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.193:34461: BlockingRecv error: recv got EOF from 127.0.192.193:34461 (error 108)
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":61,"threads_started":1}
Jun 21 07:47:56 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.194@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:47:56 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.194@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:47:56.557600 9944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:56.551916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.194:38553)
0621 07:47:56.552270 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:56.552291 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:56.552425 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:56.552730 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:56.552732 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:56.552950 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:56.553035 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:56.553040 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:56.553773 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:56.553788 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:56.554337 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:56.554344 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:56.554793 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:56.554797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:56.554894 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:56.555385 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:56.555395 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:56.557438 (+ 2043us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":80,"threads_started":1}
W20260621 07:47:56.557672 9942 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:56.552037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:38553 (local address 127.0.192.254:42251)
0621 07:47:56.552224 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:56.552229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:56.552376 (+ 147us) server_negotiation.cc:407] Connection header received
0621 07:47:56.552533 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:56.552536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:56.552604 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:56.552705 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:56.553147 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:56.553657 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:56.554467 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:56.554683 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:56.554786 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:56.557556 (+ 2770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.194:38553: BlockingRecv error: recv got EOF from 127.0.192.194:38553 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 21 07:47:56 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.195@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:47:56 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.195@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:47:56.730334 9945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:56.725820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.195:46467)
0621 07:47:56.726161 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:56.726177 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:56.726281 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:56.726486 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:56.726488 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:56.726644 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:56.726725 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:56.726730 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:56.727334 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:56.727336 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:56.727808 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:56.727815 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:56.728148 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:56.728150 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:56.728200 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:56.728624 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:56.728634 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:56.730236 (+ 1602us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":111,"threads_started":1}
W20260621 07:47:56.730469 9942 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:56.725880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:46467 (local address 127.0.192.254:42251)
0621 07:47:56.726011 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:56.726116 (+ 105us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:56.726293 (+ 177us) server_negotiation.cc:407] Connection header received
0621 07:47:56.726351 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:56.726353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:56.726395 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:56.726452 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:56.726810 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:56.727258 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:56.727892 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:56.728069 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:56.728131 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:56.730357 (+ 2226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.195:46467: BlockingRecv error: recv got EOF from 127.0.192.195:46467 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Jun 21 07:47:57 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.193@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:47:57 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.193@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:47:57.406749 9946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:57.402225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.193:50047)
0621 07:47:57.402469 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:57.402481 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:57.402577 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:57.402797 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:57.402798 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:57.402959 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:57.403040 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:57.403044 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:57.403727 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:57.403729 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:57.404199 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:57.404205 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:57.404547 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:57.404549 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:57.404626 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:57.405034 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:57.405044 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:57.406604 (+ 1560us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":88,"threads_started":1}
W20260621 07:47:57.406796 9947 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:57.402369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:50047 (local address 127.0.192.254:42251)
0621 07:47:57.402615 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:57.402619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:57.402643 (+ 24us) server_negotiation.cc:407] Connection header received
0621 07:47:57.402671 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:57.402672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:57.402708 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:57.402767 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:57.403127 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:57.403650 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:57.404284 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:57.404469 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:57.404531 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:57.406698 (+ 2167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.193:50047: BlockingRecv error: recv got EOF from 127.0.192.193:50047 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":87,"threads_started":1}
Jun 21 07:47:57 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.194@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:47:57 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.194@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:47:57.562937 9948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:57.558380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.194:58267)
0621 07:47:57.558591 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:57.558604 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:57.558699 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:57.558911 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:57.558912 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:57.559060 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:57.559144 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:57.559148 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:57.559865 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:57.559866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:57.560328 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:57.560334 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:57.560693 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:57.560695 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:57.560750 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:57.561218 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:57.561228 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:57.562842 (+ 1614us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":74,"threads_started":1}
W20260621 07:47:57.563061 9947 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:57.558430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:58267 (local address 127.0.192.254:42251)
0621 07:47:57.558560 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:57.558563 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:57.558672 (+ 109us) server_negotiation.cc:407] Connection header received
0621 07:47:57.558767 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:57.558769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:57.558807 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:57.558882 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:57.559228 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:57.559785 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:57.560417 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:57.560614 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:57.560679 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:57.562963 (+ 2284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.194:58267: BlockingRecv error: recv got EOF from 127.0.192.194:58267 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 21 07:47:57 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.195@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:47:57 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.195@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:47:57.735745 9949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:57.731089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.195:49759)
0621 07:47:57.731349 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:57.731362 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:57.731457 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:57.731664 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:57.731666 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:57.731835 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:57.731926 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:57.731930 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:57.732645 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:57.732647 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:57.733146 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:57.733153 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:57.733522 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:57.733525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:57.733578 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:57.734042 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:57.734056 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:57.735643 (+ 1587us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":91,"threads_started":1}
W20260621 07:47:57.735929 9947 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:57.731140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:49759 (local address 127.0.192.254:42251)
0621 07:47:57.731283 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:57.731288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:57.731433 (+ 145us) server_negotiation.cc:407] Connection header received
0621 07:47:57.731522 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:57.731524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:57.731567 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:57.731632 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:57.732041 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:57.732516 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:57.733235 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:57.733428 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:57.733499 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:57.735766 (+ 2267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.195:49759: BlockingRecv error: recv got EOF from 127.0.192.195:49759 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 21 07:47:58 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.193@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:47:58 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.193@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:47:58.415563 9950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:58.407584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.193:59223)
0621 07:47:58.407847 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:58.407859 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:58.407949 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:58.408196 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:58.408198 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:58.408384 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:58.408475 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:58.408481 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:58.409230 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:58.409233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:58.409749 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:58.409757 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:58.410097 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:58.410099 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:58.410150 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:58.413596 (+ 3446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:58.413608 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:58.415456 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":99,"threads_started":1}
W20260621 07:47:58.415679 9951 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:58.407639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:59223 (local address 127.0.192.254:42251)
0621 07:47:58.407851 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:58.407854 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:58.407923 (+ 69us) server_negotiation.cc:407] Connection header received
0621 07:47:58.408031 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:58.408033 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:58.408081 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:58.408151 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:58.408582 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:58.409133 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:58.409837 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:58.410018 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:58.410090 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:58.415568 (+ 5478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.193:59223: BlockingRecv error: recv got EOF from 127.0.192.193:59223 (error 108)
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":53,"threads_started":1}
Jun 21 07:47:58 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.194@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:47:58 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.194@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:47:58.569099 9952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:58.563731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.194:59935)
0621 07:47:58.564037 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:58.564053 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:58.564188 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:58.564474 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:58.564477 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:58.564690 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:58.564782 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:58.564788 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:58.565521 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:58.565525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:58.566043 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:58.566051 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:58.566497 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:58.566500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:58.566584 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:58.567119 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:58.567130 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:58.568914 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":74,"threads_started":1}
W20260621 07:47:58.569185 9951 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:58.563861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:59935 (local address 127.0.192.254:42251)
0621 07:47:58.564022 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:58.564027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:58.564140 (+ 113us) server_negotiation.cc:407] Connection header received
0621 07:47:58.564292 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:58.564297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:58.564358 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:58.564446 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:58.564903 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:58.565403 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:58.566162 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:58.566376 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:58.566495 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:58.569065 (+ 2570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.194:59935: BlockingRecv error: recv got EOF from 127.0.192.194:59935 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 21 07:47:58 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.195@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:47:58 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.195@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:47:58.741716 9953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:58.736450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.195:42939)
0621 07:47:58.736782 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:58.736798 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:58.736902 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:58.737124 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:58.737126 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:58.737328 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:58.737446 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:58.737451 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:58.738234 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:58.738236 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:58.738872 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:58.738880 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:58.739414 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:58.739424 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:58.739485 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:58.739874 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:58.739883 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:58.741584 (+ 1701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":90,"threads_started":1}
W20260621 07:47:58.741907 9951 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:58.736592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:42939 (local address 127.0.192.254:42251)
0621 07:47:58.736729 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:58.736733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:58.736875 (+ 142us) server_negotiation.cc:407] Connection header received
0621 07:47:58.736979 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:58.736981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:58.737024 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:58.737088 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:58.737553 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:58.738149 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:58.738979 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:58.739310 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:58.739402 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:58.741741 (+ 2339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.195:42939: BlockingRecv error: recv got EOF from 127.0.192.195:42939 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 21 07:47:59 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.193@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:47:59 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.193@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:47:59.421306 9954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:59.416458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.193:51955)
0621 07:47:59.416694 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:59.416705 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:59.416777 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:59.417046 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:59.417048 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:59.417215 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:59.417304 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:59.417309 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:59.418020 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:59.418022 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:59.418501 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:59.418507 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:59.418856 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:59.418858 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:59.418907 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:59.419354 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:59.419366 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:59.421124 (+ 1758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":107,"threads_started":1}
W20260621 07:47:59.421370 9955 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:59.416533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:51955 (local address 127.0.192.254:42251)
0621 07:47:59.416763 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:59.416766 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:59.416778 (+ 12us) server_negotiation.cc:407] Connection header received
0621 07:47:59.416901 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:59.416904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:59.416944 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:59.417012 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:59.417401 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:59.417920 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:59.418584 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:59.418773 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:59.418837 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:59.421253 (+ 2416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.193:51955: BlockingRecv error: recv got EOF from 127.0.192.193:51955 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":81,"threads_started":1}
Jun 21 07:47:59 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.194@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:47:59 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.194@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:47:59.575332 9956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:59.570043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.194:49953)
0621 07:47:59.570375 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:59.570387 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:59.570510 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:59.570775 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:59.570778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:59.570994 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:59.571101 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:59.571107 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:59.571854 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:59.571861 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:59.572426 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:59.572433 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:59.572890 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:59.572893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:59.572966 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:59.573413 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:59.573423 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:59.575149 (+ 1726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":100,"threads_started":1}
W20260621 07:47:59.575390 9955 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:59.570183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:49953 (local address 127.0.192.254:42251)
0621 07:47:59.570334 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:59.570339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:59.570457 (+ 118us) server_negotiation.cc:407] Connection header received
0621 07:47:59.570616 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:59.570619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:59.570668 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:59.570754 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:59.571239 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:59.571752 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:59.572546 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:59.572789 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:59.572891 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:59.575276 (+ 2385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.194:49953: BlockingRecv error: recv got EOF from 127.0.192.194:49953 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 21 07:47:59 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.195@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:47:59 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.195@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:47:59.747344 9957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:47:59.742476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.195:42693)
0621 07:47:59.742780 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0621 07:47:59.742795 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:47:59.742925 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:47:59.743229 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:47:59.743232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:47:59.743425 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:47:59.743514 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:59.743520 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:59.744278 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:59.744282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:59.744813 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:47:59.744821 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:47:59.745180 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:47:59.745183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:47:59.745236 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:59.745676 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:47:59.745685 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:47:59.747196 (+ 1511us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":91,"threads_started":1}
W20260621 07:47:59.747396 9955 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:47:59.742636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:42693 (local address 127.0.192.254:42251)
0621 07:47:59.742780 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0621 07:47:59.742785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:47:59.742870 (+ 85us) server_negotiation.cc:407] Connection header received
0621 07:47:59.743033 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:47:59.743036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:47:59.743090 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:47:59.743179 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:47:59.743616 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:59.744140 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:59.744902 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:47:59.745094 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:47:59.745165 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:47:59.747306 (+ 2141us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.195:42693: BlockingRecv error: recv got EOF from 127.0.192.195:42693 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 21 07:48:00 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.193@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:48:00 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.193@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:48:00.426578 9958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:00.422086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.193:52199)
0621 07:48:00.422359 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0621 07:48:00.422372 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:48:00.422463 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:48:00.422700 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:48:00.422702 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:48:00.422850 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:48:00.422933 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:00.422938 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:00.423552 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:00.423554 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:00.424016 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:00.424022 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:00.424361 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:00.424363 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:00.424415 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:00.424844 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:48:00.424854 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:48:00.426434 (+ 1580us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":106,"threads_started":1}
W20260621 07:48:00.426627 9959 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:48:00.422131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:52199 (local address 127.0.192.254:42251)
0621 07:48:00.422359 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:00.422363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:00.422437 (+ 74us) server_negotiation.cc:407] Connection header received
0621 07:48:00.422539 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:00.422541 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:00.422593 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:00.422671 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:48:00.423027 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:00.423474 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:00.424097 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:00.424276 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:00.424347 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:00.426526 (+ 2179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.193:52199: BlockingRecv error: recv got EOF from 127.0.192.193:52199 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":79,"threads_started":1}
Jun 21 07:48:00 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.194@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:48:00 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.194@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:48:00.581519 9960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:00.576183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.194:33103)
0621 07:48:00.576476 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0621 07:48:00.576507 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0621 07:48:00.576699 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:48:00.577007 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:48:00.577026 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:48:00.577243 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:48:00.577332 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:00.577338 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:00.578137 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:00.578143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:00.578655 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:00.578662 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:00.579075 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:00.579078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:00.579155 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:00.579591 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:48:00.579601 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:48:00.581337 (+ 1736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":85,"threads_started":1}
W20260621 07:48:00.581565 9959 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:48:00.576236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:33103 (local address 127.0.192.254:42251)
0621 07:48:00.576363 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:00.576367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:00.576600 (+ 233us) server_negotiation.cc:407] Connection header received
0621 07:48:00.576803 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:00.576806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:00.576861 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:00.576987 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:48:00.577450 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:00.578016 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:00.578772 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:00.578976 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:00.579073 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:00.581459 (+ 2386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.194:33103: BlockingRecv error: recv got EOF from 127.0.192.194:33103 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 21 07:48:00 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.195@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:48:00 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.195@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:48:00.752913 9961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:00.747930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.195:55513)
0621 07:48:00.748162 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0621 07:48:00.748174 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:48:00.748294 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:48:00.748535 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:48:00.748537 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:48:00.748740 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:48:00.748824 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:00.748829 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:00.749475 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:00.749479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:00.749970 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:00.749977 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:00.750374 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:00.750377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:00.750451 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:00.750925 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:48:00.750937 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:48:00.752769 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":71,"threads_started":1}
W20260621 07:48:00.752965 9959 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:48:00.747978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:55513 (local address 127.0.192.254:42251)
0621 07:48:00.748117 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:00.748122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:00.748246 (+ 124us) server_negotiation.cc:407] Connection header received
0621 07:48:00.748394 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:00.748397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:00.748444 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:00.748517 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:48:00.748930 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:00.749376 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:00.750076 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:00.750273 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:00.750372 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:00.752874 (+ 2502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.195:55513: BlockingRecv error: recv got EOF from 127.0.192.195:55513 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 21 07:48:01 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.193@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:48:01 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.193@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:48:01.432242 9962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:01.427330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.193:47265)
0621 07:48:01.427588 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0621 07:48:01.427601 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:48:01.427683 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:48:01.427922 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:48:01.427924 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:48:01.428072 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:48:01.428155 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:01.428159 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:01.428880 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:01.428882 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:01.429592 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:01.429602 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:01.430047 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:01.430050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:01.430124 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:01.430525 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:48:01.430535 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:48:01.432135 (+ 1600us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":95,"threads_started":1}
W20260621 07:48:01.432392 9963 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:48:01.427418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:47265 (local address 127.0.192.254:42251)
0621 07:48:01.427629 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:01.427633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:01.427645 (+ 12us) server_negotiation.cc:407] Connection header received
0621 07:48:01.427777 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:01.427779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:01.427816 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:01.427884 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:48:01.428239 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:01.428788 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:01.429698 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:01.429915 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:01.430017 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:01.432267 (+ 2250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.193:47265: BlockingRecv error: recv got EOF from 127.0.192.193:47265 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":79,"threads_started":1}
Jun 21 07:48:01 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.194@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:48:01 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.194@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:48:01.587044 9964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:01.582267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.194:34809)
0621 07:48:01.582548 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0621 07:48:01.582560 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:48:01.582653 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:48:01.582885 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:48:01.582887 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:48:01.583053 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:48:01.583140 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:01.583145 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:01.583749 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:01.583750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:01.584214 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:01.584220 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:01.584644 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:01.584646 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:01.584703 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:01.585193 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:48:01.585204 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:48:01.586898 (+ 1694us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":78,"threads_started":1}
W20260621 07:48:01.587121 9963 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:48:01.582345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:34809 (local address 127.0.192.254:42251)
0621 07:48:01.582486 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:01.582490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:01.582621 (+ 131us) server_negotiation.cc:407] Connection header received
0621 07:48:01.582717 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:01.582719 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:01.582758 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:01.582861 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:48:01.583242 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:01.583671 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:01.584318 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:01.584501 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:01.584585 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:01.586992 (+ 2407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.194:34809: BlockingRecv error: recv got EOF from 127.0.192.194:34809 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 21 07:48:01 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.195@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:48:01 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.195@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:48:01.758416 9965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:01.753704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.195:44575)
0621 07:48:01.753939 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0621 07:48:01.753950 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 07:48:01.754038 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:48:01.754234 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:48:01.754236 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:48:01.754388 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:48:01.754471 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:01.754476 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:01.755072 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:01.755074 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:01.755529 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:01.755535 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:01.756003 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:01.756007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:01.756086 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:01.756607 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:48:01.756618 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:48:01.758306 (+ 1688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":70,"threads_started":1}
W20260621 07:48:01.758534 9963 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:48:01.753760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:44575 (local address 127.0.192.254:42251)
0621 07:48:01.753897 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:01.753907 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:01.754010 (+ 103us) server_negotiation.cc:407] Connection header received
0621 07:48:01.754104 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:01.754106 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:01.754144 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:01.754205 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:48:01.754552 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:01.754994 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:01.755682 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:01.755891 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:01.755995 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:01.758431 (+ 2436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.195:44575: BlockingRecv error: recv got EOF from 127.0.192.195:44575 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 21 07:48:02 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.193@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:48:02 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.193@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:48:02.437615 9966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:02.432998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.193:48247)
0621 07:48:02.433236 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0621 07:48:02.433246 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 07:48:02.433333 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:48:02.433522 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:48:02.433524 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:48:02.433668 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:48:02.433748 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:02.433752 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:02.434356 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:02.434358 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:02.434820 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:02.434825 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:02.435182 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:02.435183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:02.435236 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:02.435658 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:48:02.435674 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:48:02.437452 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":83,"threads_started":1}
W20260621 07:48:02.437677 9967 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:48:02.433048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:48247 (local address 127.0.192.254:42251)
0621 07:48:02.433243 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:02.433246 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:02.433303 (+ 57us) server_negotiation.cc:407] Connection header received
0621 07:48:02.433405 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:02.433407 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:02.433447 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:02.433514 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:48:02.433824 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:02.434275 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:02.434898 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:02.435101 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:02.435166 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:02.437565 (+ 2399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.193:48247: BlockingRecv error: recv got EOF from 127.0.192.193:48247 (error 108)
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":73,"threads_started":1}
Jun 21 07:48:02 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.194@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:48:02 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.194@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:48:02.593734 9968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:02.588058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.194:38233)
0621 07:48:02.588375 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0621 07:48:02.588391 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:48:02.588527 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:48:02.588929 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:48:02.588931 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:48:02.589189 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:48:02.589317 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:02.589324 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:02.590149 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:02.590154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:02.590745 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:02.590753 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:02.591209 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:02.591212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:02.591298 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:02.591781 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:48:02.591792 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:48:02.593623 (+ 1831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":96,"threads_started":1}
W20260621 07:48:02.593899 9967 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:48:02.588219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:38233 (local address 127.0.192.254:42251)
0621 07:48:02.588376 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:02.588383 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:02.588488 (+ 105us) server_negotiation.cc:407] Connection header received
0621 07:48:02.588758 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:02.588762 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:02.588818 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:02.588905 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:48:02.589437 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:02.590039 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:02.590866 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:02.591098 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:02.591207 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:02.593750 (+ 2543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.194:38233: BlockingRecv error: recv got EOF from 127.0.192.194:38233 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 21 07:48:02 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.195@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:48:02 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.195@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:48:02.763789 9969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:02.759177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.195:42769)
0621 07:48:02.759408 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0621 07:48:02.759421 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:48:02.759511 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:48:02.759744 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:48:02.759746 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:48:02.759902 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:48:02.759991 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:02.759996 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:02.760645 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:02.760647 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:02.761136 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:02.761142 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:02.761490 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:02.761492 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:02.761544 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:02.761981 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:48:02.761992 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:48:02.763681 (+ 1689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":66,"threads_started":1}
W20260621 07:48:02.764007 9967 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:48:02.759226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:42769 (local address 127.0.192.254:42251)
0621 07:48:02.759344 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:02.759347 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:02.759483 (+ 136us) server_negotiation.cc:407] Connection header received
0621 07:48:02.759580 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:02.759581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:02.759621 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:02.759679 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:48:02.760080 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:02.760521 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:02.761221 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:02.761407 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:02.761476 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:02.763791 (+ 2315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.195:42769: BlockingRecv error: recv got EOF from 127.0.192.195:42769 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 21 07:48:03 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.193@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:48:03 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.193@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:48:03.448001 9970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:03.438390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.193:54973)
0621 07:48:03.438663 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0621 07:48:03.438683 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 07:48:03.438761 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:48:03.438969 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:48:03.438971 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:48:03.439126 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:48:03.439210 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:03.439214 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:03.439873 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:03.439875 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:03.440367 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:03.440374 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:03.440793 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:03.440795 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:03.440852 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:03.445920 (+ 5068us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:48:03.445930 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:48:03.447844 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":128,"threads_started":1}
W20260621 07:48:03.448155 9971 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:48:03.438521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:54973 (local address 127.0.192.254:42251)
0621 07:48:03.438779 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:03.438782 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:03.438794 (+ 12us) server_negotiation.cc:407] Connection header received
0621 07:48:03.438827 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:03.438830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:03.438868 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:03.438943 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:48:03.439311 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:03.439775 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:03.440461 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:03.440697 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:03.440771 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:03.448029 (+ 7258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.193:54973: BlockingRecv error: recv got EOF from 127.0.192.193:54973 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":77,"threads_started":1}
Jun 21 07:48:03 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.194@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:48:03 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.194@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:48:03.599773 9972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:03.594546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.194:49689)
0621 07:48:03.594788 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0621 07:48:03.594819 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0621 07:48:03.595086 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:48:03.595329 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:48:03.595332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:48:03.595524 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:48:03.595606 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:03.595611 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:03.596310 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:03.596315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:03.596854 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:03.596861 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:03.597319 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:03.597322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:03.597403 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:03.597877 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:48:03.597888 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:48:03.599674 (+ 1786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":93,"threads_started":1}
W20260621 07:48:03.600006 9971 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:48:03.594583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:49689 (local address 127.0.192.254:42251)
0621 07:48:03.594720 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:03.594723 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:03.594895 (+ 172us) server_negotiation.cc:407] Connection header received
0621 07:48:03.595179 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:03.595182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:03.595231 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:03.595309 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:48:03.595710 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:03.596201 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:03.596965 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:03.597174 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:03.597314 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:03.599822 (+ 2508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.194:49689: BlockingRecv error: recv got EOF from 127.0.192.194:49689 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 21 07:48:03 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.195@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:48:03 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.195@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:48:03.768754 9973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:03.764421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.195:33369)
0621 07:48:03.764693 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0621 07:48:03.764704 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 07:48:03.764797 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:48:03.764999 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:48:03.765001 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:48:03.765158 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:48:03.765239 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:03.765243 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:03.765846 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:03.765848 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:03.766326 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:03.766332 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:03.766676 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:03.766679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:03.766731 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:03.767134 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:48:03.767143 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:48:03.768650 (+ 1507us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":86,"threads_started":1}
W20260621 07:48:03.768862 9971 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:48:03.764457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:33369 (local address 127.0.192.254:42251)
0621 07:48:03.764598 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:03.764602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:03.764776 (+ 174us) server_negotiation.cc:407] Connection header received
0621 07:48:03.764865 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:03.764868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:03.764909 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:03.764969 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:48:03.765325 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:03.765770 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:03.766409 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:03.766592 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:03.766656 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:03.768756 (+ 2100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.195:33369: BlockingRecv error: recv got EOF from 127.0.192.195:33369 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 21 07:48:04 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.193@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:48:04 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.193@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:48:04.453438 9974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:04.448768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.193:59903)
0621 07:48:04.449037 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0621 07:48:04.449048 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 07:48:04.449121 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:48:04.449343 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:48:04.449345 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:48:04.449492 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:48:04.449574 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:04.449578 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:04.450192 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:04.450193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:04.450656 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:04.450661 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:04.450995 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:04.450997 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:04.451067 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:04.451485 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:48:04.451495 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:48:04.453307 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":75,"threads_started":1}
W20260621 07:48:04.453572 9975 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:48:04.448853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:59903 (local address 127.0.192.254:42251)
0621 07:48:04.449069 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:04.449072 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:04.449086 (+ 14us) server_negotiation.cc:407] Connection header received
0621 07:48:04.449204 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:04.449206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:04.449244 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:04.449311 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:48:04.449660 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:04.450105 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:04.450735 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:04.450917 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:04.450981 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:04.453445 (+ 2464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.193:59903: BlockingRecv error: recv got EOF from 127.0.192.193:59903 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":52,"threads_started":1}
Jun 21 07:48:04 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.194@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:48:04 dist-test-slave-lzx3 krb5kdc[8917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.192.194@KRBTEST.COM for oryx/127.0.192.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:48:04.605652 9976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:04.600548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42251 (local address 127.0.192.194:50011)
0621 07:48:04.600840 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0621 07:48:04.600852 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:48:04.600962 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:48:04.601217 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:48:04.601219 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:48:04.601407 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:48:04.601487 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:04.601491 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:04.602162 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:04.602166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:04.602769 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:04.602777 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:04.603255 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:04.603258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:04.603346 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:04.603937 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:48:04.603947 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:48:04.605540 (+ 1593us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.192.254:42251: Server oryx/127.0.192.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":93,"threads_started":1}
W20260621 07:48:04.605762 9975 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:48:04.600629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:50011 (local address 127.0.192.254:42251)
0621 07:48:04.600776 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:04.600781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:04.600919 (+ 138us) server_negotiation.cc:407] Connection header received
0621 07:48:04.601066 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:04.601068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:04.601117 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:04.601200 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:48:04.601597 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:04.602062 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:04.602892 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:04.603132 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:04.603254 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:04.605656 (+ 2402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.192.194:50011: BlockingRecv error: recv got EOF from 127.0.192.194:50011 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260621 07:48:04.671060 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 9500
I20260621 07:48:04.676402 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 9652
I20260621 07:48:04.681209 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 9784
I20260621 07:48:04.685716 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 9430
2026-06-21T07:48:04Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17663 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-21T07:48:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:48:04Z Disabled control of system clock
I20260621 07:48:04.712193 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:37535
--webserver_interface=127.0.192.254
--webserver_port=0
--builtin_ntp_servers=127.0.192.212:40457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:37535
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260621 07:48:04.805990 9980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:48:04.806174 9980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:48:04.806201 9980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:48:04.809499 9980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:48:04.809537 9980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:48:04.809551 9980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:48:04.809564 9980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:48:04.809612 9980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:48:04.813645 9980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:40457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:37535
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:37535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:48:04.814388 9980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:48:04.815428 9980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 07:48:04.822643 9980 server_base.cc:1034] running on GCE node
W20260621 07:48:04.822863 9985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:48:04.822763 9988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:48:04.822763 9986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:48:04.823511 9980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:48:04.824364 9980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:48:04.825513 9980 hybrid_clock.cc:648] HybridClock initialized: now 1782028084825492 us; error 34 us; skew 500 ppm
I20260621 07:48:04.826947 9980 webserver.cc:460] Webserver started at http://127.0.192.254:36329/ using document root <none> and password file <none>
I20260621 07:48:04.827406 9980 fs_manager.cc:359] Metadata directory not provided
I20260621 07:48:04.827455 9980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:48:04.827626 9980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:48:04.829314 9980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "133b9a2671ba4275aca457251eb7c837"
format_stamp: "Formatted at 2026-06-21 07:48:04 on dist-test-slave-lzx3"
server_key: "a0c7e9eb463c6600a11c718286822024"
server_key_iv: "4efbffc5eabefc629e312c729a4b6b28"
server_key_version: "encryptionkey@0"
I20260621 07:48:04.829777 9980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "133b9a2671ba4275aca457251eb7c837"
format_stamp: "Formatted at 2026-06-21 07:48:04 on dist-test-slave-lzx3"
server_key: "a0c7e9eb463c6600a11c718286822024"
server_key_iv: "4efbffc5eabefc629e312c729a4b6b28"
server_key_version: "encryptionkey@0"
I20260621 07:48:04.832795 9980 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:48:04.834769 9994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:48:04.835680 9980 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:48:04.835801 9980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "133b9a2671ba4275aca457251eb7c837"
format_stamp: "Formatted at 2026-06-21 07:48:04 on dist-test-slave-lzx3"
server_key: "a0c7e9eb463c6600a11c718286822024"
server_key_iv: "4efbffc5eabefc629e312c729a4b6b28"
server_key_version: "encryptionkey@0"
I20260621 07:48:04.835891 9980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:48:04.846082 9980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:48:04.846562 9980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:48:04.846719 9980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:48:04.852171 9980 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:37535
I20260621 07:48:04.852181 10046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:37535 every 8 connection(s)
I20260621 07:48:04.853011 9980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:48:04.855661 10047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:48:04.856568 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 9980
I20260621 07:48:04.856678 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:48:04.856998 771 external_mini_cluster.cc:1428] Setting key 8aedc3c16c164c2a8b365ba8aca80a0e
I20260621 07:48:04.861255 10047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 133b9a2671ba4275aca457251eb7c837: Bootstrap starting.
I20260621 07:48:04.863698 10047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 133b9a2671ba4275aca457251eb7c837: Neither blocks nor log segments found. Creating new log.
I20260621 07:48:04.864396 10047 log.cc:824] T 00000000000000000000000000000000 P 133b9a2671ba4275aca457251eb7c837: Log is configured to *not* fsync() on all Append() calls
I20260621 07:48:04.865124 10050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:04.858212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52026 (local address 127.0.192.254:37535)
0621 07:48:04.858751 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:04.858764 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:04.858793 (+ 29us) server_negotiation.cc:407] Connection header received
0621 07:48:04.859358 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:04.859372 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:04.859710 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:04.859938 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:48:04.860238 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:04.861096 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:04.862115 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:04.862621 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:04.862736 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:04.863531 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:48:04.863535 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:48:04.863545 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:48:04.863879 (+ 334us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:48:04.863923 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:48:04.863939 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:48:04.864003 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0621 07:48:04.864129 (+ 126us) server_negotiation.cc:299] Negotiation successful
0621 07:48:04.864359 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":114,"threads_started":1}
I20260621 07:48:04.866123 10047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 133b9a2671ba4275aca457251eb7c837: No bootstrap required, opened a new log
I20260621 07:48:04.869385 10047 raft_consensus.cc:348] T 00000000000000000000000000000000 P 133b9a2671ba4275aca457251eb7c837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133b9a2671ba4275aca457251eb7c837" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 37535 } }
I20260621 07:48:04.869575 10047 raft_consensus.cc:374] T 00000000000000000000000000000000 P 133b9a2671ba4275aca457251eb7c837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:48:04.869632 10047 raft_consensus.cc:729] T 00000000000000000000000000000000 P 133b9a2671ba4275aca457251eb7c837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 133b9a2671ba4275aca457251eb7c837, State: Initialized, Role: FOLLOWER
I20260621 07:48:04.870011 10047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 133b9a2671ba4275aca457251eb7c837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133b9a2671ba4275aca457251eb7c837" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 37535 } }
I20260621 07:48:04.870134 10047 raft_consensus.cc:388] T 00000000000000000000000000000000 P 133b9a2671ba4275aca457251eb7c837 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:48:04.870189 10047 raft_consensus.cc:482] T 00000000000000000000000000000000 P 133b9a2671ba4275aca457251eb7c837 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:48:04.870268 10047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 133b9a2671ba4275aca457251eb7c837 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:48:04.870958 10047 raft_consensus.cc:504] T 00000000000000000000000000000000 P 133b9a2671ba4275aca457251eb7c837 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133b9a2671ba4275aca457251eb7c837" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 37535 } }
I20260621 07:48:04.871240 10047 leader_election.cc:302] T 00000000000000000000000000000000 P 133b9a2671ba4275aca457251eb7c837 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 133b9a2671ba4275aca457251eb7c837; no voters:
I20260621 07:48:04.871503 10047 leader_election.cc:288] T 00000000000000000000000000000000 P 133b9a2671ba4275aca457251eb7c837 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:48:04.871588 10052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 133b9a2671ba4275aca457251eb7c837 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:48:04.871778 10052 raft_consensus.cc:686] T 00000000000000000000000000000000 P 133b9a2671ba4275aca457251eb7c837 [term 1 LEADER]: Becoming Leader. State: Replica: 133b9a2671ba4275aca457251eb7c837, State: Running, Role: LEADER
I20260621 07:48:04.872107 10052 consensus_queue.cc:237] T 00000000000000000000000000000000 P 133b9a2671ba4275aca457251eb7c837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133b9a2671ba4275aca457251eb7c837" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 37535 } }
I20260621 07:48:04.872704 10047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 133b9a2671ba4275aca457251eb7c837 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:48:04.873193 10052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 133b9a2671ba4275aca457251eb7c837 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 133b9a2671ba4275aca457251eb7c837. Latest consensus state: current_term: 1 leader_uuid: "133b9a2671ba4275aca457251eb7c837" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133b9a2671ba4275aca457251eb7c837" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 37535 } } }
I20260621 07:48:04.873312 10052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 133b9a2671ba4275aca457251eb7c837 [sys.catalog]: This master's current role is: LEADER
I20260621 07:48:04.873533 10053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 133b9a2671ba4275aca457251eb7c837 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "133b9a2671ba4275aca457251eb7c837" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133b9a2671ba4275aca457251eb7c837" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 37535 } } }
I20260621 07:48:04.873620 10053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 133b9a2671ba4275aca457251eb7c837 [sys.catalog]: This master's current role is: LEADER
I20260621 07:48:04.873968 10057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:48:04.876963 10057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:48:04.881645 10057 catalog_manager.cc:1347] Generated new cluster ID: bcd151f6bdd541529a65c47902e2301c
I20260621 07:48:04.881702 10057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:48:04.890841 10057 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:48:04.891842 10057 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:48:04.896438 10057 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 133b9a2671ba4275aca457251eb7c837: Generated new TSK 0
I20260621 07:48:04.897071 10057 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:48:04.908812 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.193:0
--local_ip_for_outbound_sockets=127.0.192.193
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:37535
--builtin_ntp_servers=127.0.192.212:40457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260621 07:48:05.011637 10071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:48:05.011971 10071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:48:05.012027 10071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:48:05.018800 10071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:48:05.018865 10071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:48:05.018941 10071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.193
I20260621 07:48:05.026846 10071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:40457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:37535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.193
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:48:05.028170 10071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:48:05.029762 10071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:48:05.040903 10079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:48:05.041280 10071 server_base.cc:1034] running on GCE node
W20260621 07:48:05.041340 10077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:48:05.041568 10076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:48:05.041939 10071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:48:05.042515 10071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:48:05.043690 10071 hybrid_clock.cc:648] HybridClock initialized: now 1782028085043662 us; error 47 us; skew 500 ppm
I20260621 07:48:05.045302 10071 webserver.cc:460] Webserver started at http://127.0.192.193:37207/ using document root <none> and password file <none>
I20260621 07:48:05.045809 10071 fs_manager.cc:359] Metadata directory not provided
I20260621 07:48:05.045887 10071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:48:05.046082 10071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:48:05.047729 10071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-0/data/instance:
uuid: "61de2e8360e14a70b36bc39d02c0d31b"
format_stamp: "Formatted at 2026-06-21 07:48:05 on dist-test-slave-lzx3"
server_key: "bd2ce7136ff863a0923027e3a8244123"
server_key_iv: "66e54c128bfaa140327e4022dd6c5f26"
server_key_version: "encryptionkey@0"
I20260621 07:48:05.048206 10071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-0/wal/instance:
uuid: "61de2e8360e14a70b36bc39d02c0d31b"
format_stamp: "Formatted at 2026-06-21 07:48:05 on dist-test-slave-lzx3"
server_key: "bd2ce7136ff863a0923027e3a8244123"
server_key_iv: "66e54c128bfaa140327e4022dd6c5f26"
server_key_version: "encryptionkey@0"
I20260621 07:48:05.051275 10071 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 07:48:05.053532 10085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:48:05.054622 10071 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260621 07:48:05.054702 10071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-0/wal
uuid: "61de2e8360e14a70b36bc39d02c0d31b"
format_stamp: "Formatted at 2026-06-21 07:48:05 on dist-test-slave-lzx3"
server_key: "bd2ce7136ff863a0923027e3a8244123"
server_key_iv: "66e54c128bfaa140327e4022dd6c5f26"
server_key_version: "encryptionkey@0"
I20260621 07:48:05.054772 10071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:48:05.066639 10071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:48:05.067077 10071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:48:05.067198 10071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:48:05.067579 10071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:48:05.068262 10071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:48:05.068301 10071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:48:05.068331 10071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:48:05.068344 10071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:48:05.076005 10071 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.193:36033
I20260621 07:48:05.076071 10198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.193:36033 every 8 connection(s)
I20260621 07:48:05.076786 10071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
I20260621 07:48:05.084301 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 10071
I20260621 07:48:05.084321 10050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:05.078304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:51005 (local address 127.0.192.254:37535)
0621 07:48:05.078448 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:05.078453 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:05.079332 (+ 879us) server_negotiation.cc:407] Connection header received
0621 07:48:05.080422 (+ 1090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:05.080424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:05.080462 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:05.080527 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:48:05.081275 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:05.081739 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:05.082732 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:05.082947 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:05.083024 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:05.083894 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:48:05.083896 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:48:05.083898 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:48:05.083908 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:48:05.083922 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:48:05.083928 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:48:05.083980 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0621 07:48:05.084120 (+ 140us) server_negotiation.cc:299] Negotiation successful
0621 07:48:05.084179 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 07:48:05.084409 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-0/wal/instance
I20260621 07:48:05.084709 771 external_mini_cluster.cc:1428] Setting key 9706cd3945d2498ab81a0dc9820e6b09
I20260621 07:48:05.084981 10201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:05.078672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:37535 (local address 127.0.192.193:51005)
0621 07:48:05.079211 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0621 07:48:05.079243 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0621 07:48:05.080270 (+ 1027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:48:05.080724 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:48:05.080729 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:48:05.080798 (+ 69us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:48:05.081169 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:05.081179 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:05.081831 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:05.081833 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:05.082617 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:05.082625 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:05.083065 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:05.083067 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:05.083192 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:05.083695 (+ 503us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:48:05.083696 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:48:05.083726 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:48:05.083775 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:48:05.083991 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:48:05.083996 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:48:05.084000 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 07:48:05.084076 (+ 76us) client_negotiation.cc:240] Negotiation successful
0621 07:48:05.084247 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":99,"threads_started":1}
I20260621 07:48:05.086090 10199 heartbeater.cc:347] Connected to a master server at 127.0.192.254:37535
I20260621 07:48:05.086273 10199 heartbeater.cc:464] Registering TS with master...
I20260621 07:48:05.086462 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.194:0
--local_ip_for_outbound_sockets=127.0.192.194
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:37535
--builtin_ntp_servers=127.0.192.212:40457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260621 07:48:05.086740 10199 heartbeater.cc:511] Master 127.0.192.254:37535 requested a full tablet report, sending...
I20260621 07:48:05.088110 10011 ts_manager.cc:194] Registered new tserver with Master: 61de2e8360e14a70b36bc39d02c0d31b (127.0.192.193:36033)
I20260621 07:48:05.089507 10011 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.192.193:51005
W20260621 07:48:05.180902 10202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:48:05.181087 10202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:48:05.181118 10202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:48:05.184747 10202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:48:05.184785 10202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:48:05.184830 10202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.194
I20260621 07:48:05.189103 10202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:40457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:37535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.194
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:48:05.189849 10202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:48:05.190934 10202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:48:05.198921 10208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:48:05.198925 10210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:48:05.199018 10207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:48:05.199128 10202 server_base.cc:1034] running on GCE node
I20260621 07:48:05.199460 10202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:48:05.199946 10202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:48:05.201112 10202 hybrid_clock.cc:648] HybridClock initialized: now 1782028085201105 us; error 25 us; skew 500 ppm
I20260621 07:48:05.202631 10202 webserver.cc:460] Webserver started at http://127.0.192.194:40835/ using document root <none> and password file <none>
I20260621 07:48:05.203137 10202 fs_manager.cc:359] Metadata directory not provided
I20260621 07:48:05.203233 10202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:48:05.203438 10202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:48:05.205196 10202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-1/data/instance:
uuid: "9d583b23463144f0a0f06248f59a2df1"
format_stamp: "Formatted at 2026-06-21 07:48:05 on dist-test-slave-lzx3"
server_key: "29945024d6dbd9e37ffcb3bd08622c9c"
server_key_iv: "88112860a13de3baf19c049ebed2b1c6"
server_key_version: "encryptionkey@0"
I20260621 07:48:05.205690 10202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-1/wal/instance:
uuid: "9d583b23463144f0a0f06248f59a2df1"
format_stamp: "Formatted at 2026-06-21 07:48:05 on dist-test-slave-lzx3"
server_key: "29945024d6dbd9e37ffcb3bd08622c9c"
server_key_iv: "88112860a13de3baf19c049ebed2b1c6"
server_key_version: "encryptionkey@0"
I20260621 07:48:05.208659 10202 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:48:05.210690 10216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:48:05.211753 10202 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:48:05.211884 10202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-1/wal
uuid: "9d583b23463144f0a0f06248f59a2df1"
format_stamp: "Formatted at 2026-06-21 07:48:05 on dist-test-slave-lzx3"
server_key: "29945024d6dbd9e37ffcb3bd08622c9c"
server_key_iv: "88112860a13de3baf19c049ebed2b1c6"
server_key_version: "encryptionkey@0"
I20260621 07:48:05.211979 10202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:48:05.234930 10202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:48:05.235540 10202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:48:05.235714 10202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:48:05.236354 10202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:48:05.237190 10202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:48:05.237237 10202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:48:05.237272 10202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:48:05.237290 10202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:48:05.245832 10202 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.194:38863
I20260621 07:48:05.245863 10329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.194:38863 every 8 connection(s)
I20260621 07:48:05.246748 10202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
I20260621 07:48:05.251633 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 10202
I20260621 07:48:05.251750 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-1/wal/instance
I20260621 07:48:05.252020 771 external_mini_cluster.cc:1428] Setting key 03be7a0efcf1f3c955d69997224806b6
I20260621 07:48:05.253993 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.195:0
--local_ip_for_outbound_sockets=127.0.192.195
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:37535
--builtin_ntp_servers=127.0.192.212:40457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260621 07:48:05.255013 10050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:05.248135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:34295 (local address 127.0.192.254:37535)
0621 07:48:05.248259 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:05.248269 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:05.249136 (+ 867us) server_negotiation.cc:407] Connection header received
0621 07:48:05.250353 (+ 1217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:05.250356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:05.250410 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:05.250508 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:48:05.251245 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:05.251694 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:05.253093 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:05.253413 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:05.253505 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:05.254620 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:48:05.254623 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:48:05.254624 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:48:05.254635 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:48:05.254651 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:48:05.254658 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:48:05.254711 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0621 07:48:05.254873 (+ 162us) server_negotiation.cc:299] Negotiation successful
0621 07:48:05.254918 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260621 07:48:05.255828 10332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:05.248407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:37535 (local address 127.0.192.194:34295)
0621 07:48:05.249004 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0621 07:48:05.249036 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0621 07:48:05.250153 (+ 1117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:48:05.250671 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:48:05.250677 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:48:05.250743 (+ 66us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:48:05.251116 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:05.251125 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:05.251820 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:05.251825 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:05.252968 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:05.252977 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:05.253564 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:05.253580 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:05.253756 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:05.254357 (+ 601us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:48:05.254361 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:48:05.254409 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:48:05.254480 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:48:05.254736 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:48:05.254745 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:48:05.254753 (+ 8us) client_negotiation.cc:769] Sending connection context
0621 07:48:05.254863 (+ 110us) client_negotiation.cc:240] Negotiation successful
0621 07:48:05.255107 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":135,"threads_started":1}
I20260621 07:48:05.256735 10330 heartbeater.cc:347] Connected to a master server at 127.0.192.254:37535
I20260621 07:48:05.256942 10330 heartbeater.cc:464] Registering TS with master...
I20260621 07:48:05.257328 10330 heartbeater.cc:511] Master 127.0.192.254:37535 requested a full tablet report, sending...
I20260621 07:48:05.258291 10011 ts_manager.cc:194] Registered new tserver with Master: 9d583b23463144f0a0f06248f59a2df1 (127.0.192.194:38863)
I20260621 07:48:05.258844 10011 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.192.194:34295
W20260621 07:48:05.349632 10333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:48:05.349828 10333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:48:05.349884 10333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:48:05.353359 10333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:48:05.353406 10333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:48:05.353503 10333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.195
I20260621 07:48:05.357515 10333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:40457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:37535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.195
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:48:05.358395 10333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:48:05.359530 10333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:48:05.367712 10338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:48:05.367762 10339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:48:05.367717 10341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:48:05.367919 10333 server_base.cc:1034] running on GCE node
I20260621 07:48:05.368271 10333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:48:05.368827 10333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:48:05.369969 10333 hybrid_clock.cc:648] HybridClock initialized: now 1782028085369960 us; error 24 us; skew 500 ppm
I20260621 07:48:05.371513 10333 webserver.cc:460] Webserver started at http://127.0.192.195:41953/ using document root <none> and password file <none>
I20260621 07:48:05.372022 10333 fs_manager.cc:359] Metadata directory not provided
I20260621 07:48:05.372078 10333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:48:05.372313 10333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:48:05.374305 10333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-2/data/instance:
uuid: "8c5cab0c95fe458b8dcb4ed3b5eaf079"
format_stamp: "Formatted at 2026-06-21 07:48:05 on dist-test-slave-lzx3"
server_key: "bbb986fed84f94051c1d2ed061673a6f"
server_key_iv: "2e3c5aaadbba711bd1e5fbe7bf7e9c28"
server_key_version: "encryptionkey@0"
I20260621 07:48:05.374830 10333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-2/wal/instance:
uuid: "8c5cab0c95fe458b8dcb4ed3b5eaf079"
format_stamp: "Formatted at 2026-06-21 07:48:05 on dist-test-slave-lzx3"
server_key: "bbb986fed84f94051c1d2ed061673a6f"
server_key_iv: "2e3c5aaadbba711bd1e5fbe7bf7e9c28"
server_key_version: "encryptionkey@0"
I20260621 07:48:05.377818 10333 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:48:05.379889 10347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:48:05.380903 10333 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260621 07:48:05.381003 10333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-2/wal
uuid: "8c5cab0c95fe458b8dcb4ed3b5eaf079"
format_stamp: "Formatted at 2026-06-21 07:48:05 on dist-test-slave-lzx3"
server_key: "bbb986fed84f94051c1d2ed061673a6f"
server_key_iv: "2e3c5aaadbba711bd1e5fbe7bf7e9c28"
server_key_version: "encryptionkey@0"
I20260621 07:48:05.381086 10333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:48:05.405071 10333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:48:05.405550 10333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:48:05.405720 10333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:48:05.406154 10333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:48:05.406867 10333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:48:05.406905 10333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:48:05.406960 10333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:48:05.406994 10333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:48:05.414723 10333 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.195:41301
I20260621 07:48:05.414759 10460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.195:41301 every 8 connection(s)
I20260621 07:48:05.415396 10333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
I20260621 07:48:05.419548 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 10333
I20260621 07:48:05.419669 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782028028522580-771-0/minicluster-data/ts-2/wal/instance
I20260621 07:48:05.419981 771 external_mini_cluster.cc:1428] Setting key 9193acd4f265be2f363704fa4b4d1045
I20260621 07:48:05.423558 10050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:05.416878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:37475 (local address 127.0.192.254:37535)
0621 07:48:05.416982 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:05.416990 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:05.417806 (+ 816us) server_negotiation.cc:407] Connection header received
0621 07:48:05.418956 (+ 1150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:05.418959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:05.419002 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:05.419088 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:48:05.419906 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:05.420607 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:05.422044 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:05.422253 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:05.422326 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:05.423210 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:48:05.423213 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:48:05.423215 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:48:05.423225 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:48:05.423248 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:48:05.423255 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:48:05.423314 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0621 07:48:05.423419 (+ 105us) server_negotiation.cc:299] Negotiation successful
0621 07:48:05.423460 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260621 07:48:05.424264 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:05.417138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:37535 (local address 127.0.192.195:37475)
0621 07:48:05.417686 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0621 07:48:05.417717 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0621 07:48:05.418773 (+ 1056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:48:05.419218 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:48:05.419224 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:48:05.419294 (+ 70us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:48:05.419714 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:05.419747 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:05.420707 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:05.420713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:05.421920 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:05.421928 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:05.422371 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:05.422373 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:05.422518 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:05.422981 (+ 463us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:48:05.422983 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:48:05.423016 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:48:05.423076 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:48:05.423312 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:48:05.423317 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:48:05.423319 (+ 2us) client_negotiation.cc:769] Sending connection context
0621 07:48:05.423399 (+ 80us) client_negotiation.cc:240] Negotiation successful
0621 07:48:05.423578 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":95,"threads_started":1}
I20260621 07:48:05.425189 10461 heartbeater.cc:347] Connected to a master server at 127.0.192.254:37535
I20260621 07:48:05.425355 10461 heartbeater.cc:464] Registering TS with master...
I20260621 07:48:05.425741 10461 heartbeater.cc:511] Master 127.0.192.254:37535 requested a full tablet report, sending...
I20260621 07:48:05.426491 10011 ts_manager.cc:194] Registered new tserver with Master: 8c5cab0c95fe458b8dcb4ed3b5eaf079 (127.0.192.195:41301)
I20260621 07:48:05.426915 10011 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.192.195:37475
I20260621 07:48:05.433298 771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260621 07:48:05.435490 10050 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:48:05.434830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52032 (local address 127.0.192.254:37535)
0621 07:48:05.434918 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:05.434921 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:05.435063 (+ 142us) server_negotiation.cc:407] Connection header received
0621 07:48:05.435135 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:05.435137 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:05.435166 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:05.435227 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:48:05.435387 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52032: BlockingRecv error: recv got EOF from 127.0.0.1:52032 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260621 07:48:05.435834 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 10071
I20260621 07:48:05.440810 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 10202
I20260621 07:48:05.445995 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 10333
I20260621 07:48:05.450649 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 9980
2026-06-21T07:48:05Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (765 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-21T07:48:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:48:05Z Disabled control of system clock
I20260621 07:48:05.478464 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:34071
--webserver_interface=127.0.192.254
--webserver_port=0
--builtin_ntp_servers=127.0.192.212:35023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:34071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260621 07:48:05.582602 10472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:48:05.582839 10472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:48:05.582875 10472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:48:05.586678 10472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:48:05.586721 10472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:48:05.586758 10472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:48:05.586774 10472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:48:05.586788 10472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:48:05.591266 10472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:35023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:34071
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:34071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:48:05.592167 10472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:48:05.593485 10472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:48:05.601810 10477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:48:05.601869 10478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:48:05.601810 10480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:48:05.602159 10472 server_base.cc:1034] running on GCE node
I20260621 07:48:05.602716 10472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:48:05.603753 10472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:48:05.604938 10472 hybrid_clock.cc:648] HybridClock initialized: now 1782028085604904 us; error 43 us; skew 500 ppm
I20260621 07:48:05.606731 10472 webserver.cc:460] Webserver started at http://127.0.192.254:34113/ using document root <none> and password file <none>
I20260621 07:48:05.607276 10472 fs_manager.cc:359] Metadata directory not provided
I20260621 07:48:05.607337 10472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:48:05.607496 10472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:48:05.609521 10472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "41f9fa1a5fb441419f9996351240dfdf"
format_stamp: "Formatted at 2026-06-21 07:48:05 on dist-test-slave-lzx3"
server_key: "3690f9cb5328558db9dbde722129ca85"
server_key_iv: "8ee2a96e7548f511036879eb24f9ba6f"
server_key_version: "encryptionkey@0"
I20260621 07:48:05.610026 10472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "41f9fa1a5fb441419f9996351240dfdf"
format_stamp: "Formatted at 2026-06-21 07:48:05 on dist-test-slave-lzx3"
server_key: "3690f9cb5328558db9dbde722129ca85"
server_key_iv: "8ee2a96e7548f511036879eb24f9ba6f"
server_key_version: "encryptionkey@0"
I20260621 07:48:05.613497 10472 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:48:05.615729 10486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:48:05.616979 10472 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260621 07:48:05.617106 10472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "41f9fa1a5fb441419f9996351240dfdf"
format_stamp: "Formatted at 2026-06-21 07:48:05 on dist-test-slave-lzx3"
server_key: "3690f9cb5328558db9dbde722129ca85"
server_key_iv: "8ee2a96e7548f511036879eb24f9ba6f"
server_key_version: "encryptionkey@0"
I20260621 07:48:05.617213 10472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:48:05.621587 10472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:48:05.622148 10472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:48:05.622327 10472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:48:05.628826 10472 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:34071
I20260621 07:48:05.628842 10538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:34071 every 8 connection(s)
I20260621 07:48:05.629696 10472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:48:05.632714 10539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:48:05.633033 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 10472
I20260621 07:48:05.633170 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:48:05.633459 771 external_mini_cluster.cc:1428] Setting key 1cbad3e179027fa793f1f4580b03e0af
I20260621 07:48:05.638615 10539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41f9fa1a5fb441419f9996351240dfdf: Bootstrap starting.
I20260621 07:48:05.638818 10542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:05.634824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47834 (local address 127.0.192.254:34071)
0621 07:48:05.635407 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:05.635421 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:05.635454 (+ 33us) server_negotiation.cc:407] Connection header received
0621 07:48:05.636093 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:05.636108 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:05.636432 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:05.636833 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:48:05.637608 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:48:05.637612 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:48:05.637624 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:48:05.637634 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:48:05.637685 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:48:05.637700 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:48:05.637854 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0621 07:48:05.637922 (+ 68us) server_negotiation.cc:299] Negotiation successful
0621 07:48:05.638145 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":113,"threads_started":1}
I20260621 07:48:05.641023 10539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41f9fa1a5fb441419f9996351240dfdf: Neither blocks nor log segments found. Creating new log.
I20260621 07:48:05.641693 10539 log.cc:824] T 00000000000000000000000000000000 P 41f9fa1a5fb441419f9996351240dfdf: Log is configured to *not* fsync() on all Append() calls
I20260621 07:48:05.643384 10539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41f9fa1a5fb441419f9996351240dfdf: No bootstrap required, opened a new log
I20260621 07:48:05.646662 10539 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41f9fa1a5fb441419f9996351240dfdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f9fa1a5fb441419f9996351240dfdf" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34071 } }
I20260621 07:48:05.646898 10539 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41f9fa1a5fb441419f9996351240dfdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:48:05.646996 10539 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41f9fa1a5fb441419f9996351240dfdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41f9fa1a5fb441419f9996351240dfdf, State: Initialized, Role: FOLLOWER
I20260621 07:48:05.647444 10539 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41f9fa1a5fb441419f9996351240dfdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f9fa1a5fb441419f9996351240dfdf" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34071 } }
I20260621 07:48:05.647600 10539 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41f9fa1a5fb441419f9996351240dfdf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:48:05.647689 10539 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41f9fa1a5fb441419f9996351240dfdf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:48:05.647787 10539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41f9fa1a5fb441419f9996351240dfdf [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:48:05.648646 10539 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41f9fa1a5fb441419f9996351240dfdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f9fa1a5fb441419f9996351240dfdf" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34071 } }
I20260621 07:48:05.648976 10539 leader_election.cc:302] T 00000000000000000000000000000000 P 41f9fa1a5fb441419f9996351240dfdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41f9fa1a5fb441419f9996351240dfdf; no voters:
I20260621 07:48:05.649269 10539 leader_election.cc:288] T 00000000000000000000000000000000 P 41f9fa1a5fb441419f9996351240dfdf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:48:05.649348 10544 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41f9fa1a5fb441419f9996351240dfdf [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:48:05.649565 10544 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41f9fa1a5fb441419f9996351240dfdf [term 1 LEADER]: Becoming Leader. State: Replica: 41f9fa1a5fb441419f9996351240dfdf, State: Running, Role: LEADER
I20260621 07:48:05.649847 10544 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41f9fa1a5fb441419f9996351240dfdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f9fa1a5fb441419f9996351240dfdf" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34071 } }
I20260621 07:48:05.650331 10539 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41f9fa1a5fb441419f9996351240dfdf [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:48:05.651341 10546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41f9fa1a5fb441419f9996351240dfdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41f9fa1a5fb441419f9996351240dfdf. Latest consensus state: current_term: 1 leader_uuid: "41f9fa1a5fb441419f9996351240dfdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f9fa1a5fb441419f9996351240dfdf" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34071 } } }
I20260621 07:48:05.651470 10546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41f9fa1a5fb441419f9996351240dfdf [sys.catalog]: This master's current role is: LEADER
I20260621 07:48:05.651723 10545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41f9fa1a5fb441419f9996351240dfdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41f9fa1a5fb441419f9996351240dfdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f9fa1a5fb441419f9996351240dfdf" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34071 } } }
I20260621 07:48:05.651808 10553 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:48:05.651826 10545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41f9fa1a5fb441419f9996351240dfdf [sys.catalog]: This master's current role is: LEADER
I20260621 07:48:05.654822 10553 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:48:05.660223 10553 catalog_manager.cc:1347] Generated new cluster ID: 40d1febf18994ca8aad2a0dd432b2f7d
I20260621 07:48:05.660313 10553 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:48:05.670856 10553 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:48:05.671131 10553 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:48:05.686384 10553 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41f9fa1a5fb441419f9996351240dfdf: Generated new TSK 0
I20260621 07:48:05.687193 10553 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:48:05.696233 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.193:0
--local_ip_for_outbound_sockets=127.0.192.193
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:34071
--builtin_ntp_servers=127.0.192.212:35023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260621 07:48:05.802016 10563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:48:05.802237 10563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:48:05.802274 10563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:48:05.806253 10563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:48:05.806304 10563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:48:05.806358 10563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.193
I20260621 07:48:05.811043 10563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:35023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:34071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.193
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:48:05.812008 10563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:48:05.813347 10563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:48:05.822367 10569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:48:05.822427 10568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:48:05.822431 10571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:48:05.822590 10563 server_base.cc:1034] running on GCE node
I20260621 07:48:05.823094 10563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:48:05.823707 10563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:48:05.824903 10563 hybrid_clock.cc:648] HybridClock initialized: now 1782028085824896 us; error 32 us; skew 500 ppm
I20260621 07:48:05.826555 10563 webserver.cc:460] Webserver started at http://127.0.192.193:43071/ using document root <none> and password file <none>
I20260621 07:48:05.827189 10563 fs_manager.cc:359] Metadata directory not provided
I20260621 07:48:05.827399 10563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:48:05.827677 10563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:48:05.829775 10563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-0/data/instance:
uuid: "b6ead24982cd4ddc885911693157bb48"
format_stamp: "Formatted at 2026-06-21 07:48:05 on dist-test-slave-lzx3"
server_key: "7b22288a4ed046e4fd30b53836d0d32d"
server_key_iv: "25e0e4ddf2fffac3de1ecc1cc45e1692"
server_key_version: "encryptionkey@0"
I20260621 07:48:05.830330 10563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-0/wal/instance:
uuid: "b6ead24982cd4ddc885911693157bb48"
format_stamp: "Formatted at 2026-06-21 07:48:05 on dist-test-slave-lzx3"
server_key: "7b22288a4ed046e4fd30b53836d0d32d"
server_key_iv: "25e0e4ddf2fffac3de1ecc1cc45e1692"
server_key_version: "encryptionkey@0"
I20260621 07:48:05.833789 10563 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:48:05.836017 10577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:48:05.837179 10563 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260621 07:48:05.837308 10563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-0/wal
uuid: "b6ead24982cd4ddc885911693157bb48"
format_stamp: "Formatted at 2026-06-21 07:48:05 on dist-test-slave-lzx3"
server_key: "7b22288a4ed046e4fd30b53836d0d32d"
server_key_iv: "25e0e4ddf2fffac3de1ecc1cc45e1692"
server_key_version: "encryptionkey@0"
I20260621 07:48:05.837420 10563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:48:05.841758 10563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:48:05.842307 10563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:48:05.842495 10563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:48:05.843006 10563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:48:05.843832 10563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:48:05.843901 10563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:48:05.843966 10563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:48:05.844014 10563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:48:05.852460 10563 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.193:44519
I20260621 07:48:05.852548 10690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.193:44519 every 8 connection(s)
I20260621 07:48:05.853394 10563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
I20260621 07:48:05.858979 10542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:05.855085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:40895 (local address 127.0.192.254:34071)
0621 07:48:05.855241 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:05.855246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:05.856178 (+ 932us) server_negotiation.cc:407] Connection header received
0621 07:48:05.857471 (+ 1293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:05.857473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:05.857519 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:05.857599 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:48:05.858598 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:48:05.858600 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:48:05.858602 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:48:05.858612 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:48:05.858630 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:48:05.858638 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:48:05.858708 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 07:48:05.858845 (+ 137us) server_negotiation.cc:299] Negotiation successful
0621 07:48:05.858878 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 07:48:05.859767 10693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:05.855395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:34071 (local address 127.0.192.193:40895)
0621 07:48:05.856036 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0621 07:48:05.856077 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 07:48:05.857288 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:48:05.857780 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:48:05.857786 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:48:05.857870 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:48:05.858369 (+ 499us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:48:05.858370 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:48:05.858406 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:48:05.858469 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:48:05.858713 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:48:05.858719 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:48:05.858723 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 07:48:05.858868 (+ 145us) client_negotiation.cc:240] Negotiation successful
0621 07:48:05.859071 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":99,"threads_started":1}
I20260621 07:48:05.861150 10691 heartbeater.cc:347] Connected to a master server at 127.0.192.254:34071
I20260621 07:48:05.861407 10691 heartbeater.cc:464] Registering TS with master...
I20260621 07:48:05.861801 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 10563
I20260621 07:48:05.861871 10691 heartbeater.cc:511] Master 127.0.192.254:34071 requested a full tablet report, sending...
I20260621 07:48:05.861928 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-0/wal/instance
I20260621 07:48:05.862241 771 external_mini_cluster.cc:1428] Setting key 510802a064fa6cced71a9f121cfaf907
I20260621 07:48:05.863372 10503 ts_manager.cc:194] Registered new tserver with Master: b6ead24982cd4ddc885911693157bb48 (127.0.192.193:44519)
I20260621 07:48:05.864315 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.194:0
--local_ip_for_outbound_sockets=127.0.192.194
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:34071
--builtin_ntp_servers=127.0.192.212:35023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260621 07:48:05.969693 10694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:48:05.969970 10694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:48:05.970046 10694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:48:05.974035 10694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:48:05.974108 10694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:48:05.974195 10694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.194
I20260621 07:48:05.978787 10694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:35023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:34071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.194
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:48:05.979782 10694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:48:05.981129 10694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:48:05.990101 10702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:48:05.990113 10699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:48:05.990113 10700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:48:05.990429 10694 server_base.cc:1034] running on GCE node
I20260621 07:48:05.990833 10694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:48:05.991358 10694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:48:05.992543 10694 hybrid_clock.cc:648] HybridClock initialized: now 1782028085992512 us; error 41 us; skew 500 ppm
I20260621 07:48:05.994377 10694 webserver.cc:460] Webserver started at http://127.0.192.194:34417/ using document root <none> and password file <none>
I20260621 07:48:05.994921 10694 fs_manager.cc:359] Metadata directory not provided
I20260621 07:48:05.994982 10694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:48:05.995226 10694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:48:05.997208 10694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-1/data/instance:
uuid: "bedf72935d114b66b62e957a90f3cb40"
format_stamp: "Formatted at 2026-06-21 07:48:05 on dist-test-slave-lzx3"
server_key: "4d7f485480073977d22237a349ac2df6"
server_key_iv: "78946e450cd279f36c048488155efc69"
server_key_version: "encryptionkey@0"
I20260621 07:48:05.997756 10694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-1/wal/instance:
uuid: "bedf72935d114b66b62e957a90f3cb40"
format_stamp: "Formatted at 2026-06-21 07:48:05 on dist-test-slave-lzx3"
server_key: "4d7f485480073977d22237a349ac2df6"
server_key_iv: "78946e450cd279f36c048488155efc69"
server_key_version: "encryptionkey@0"
I20260621 07:48:06.001129 10694 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:48:06.003338 10708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:48:06.004475 10694 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260621 07:48:06.004626 10694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-1/wal
uuid: "bedf72935d114b66b62e957a90f3cb40"
format_stamp: "Formatted at 2026-06-21 07:48:05 on dist-test-slave-lzx3"
server_key: "4d7f485480073977d22237a349ac2df6"
server_key_iv: "78946e450cd279f36c048488155efc69"
server_key_version: "encryptionkey@0"
I20260621 07:48:06.004730 10694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:48:06.008981 10694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:48:06.009518 10694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:48:06.009688 10694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:48:06.010190 10694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:48:06.011022 10694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:48:06.011067 10694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:48:06.011132 10694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:48:06.011168 10694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:48:06.019665 10694 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.194:38079
I20260621 07:48:06.019706 10821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.194:38079 every 8 connection(s)
I20260621 07:48:06.020538 10694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
I20260621 07:48:06.026176 10542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:06.022309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:48697 (local address 127.0.192.254:34071)
0621 07:48:06.022482 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:06.022487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:06.023477 (+ 990us) server_negotiation.cc:407] Connection header received
0621 07:48:06.024726 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:06.024729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:06.024764 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:06.024838 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:48:06.025796 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:48:06.025799 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:48:06.025801 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:48:06.025812 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:48:06.025842 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:48:06.025850 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:48:06.025929 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0621 07:48:06.026043 (+ 114us) server_negotiation.cc:299] Negotiation successful
0621 07:48:06.026076 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260621 07:48:06.026954 10824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:06.022721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:34071 (local address 127.0.192.194:48697)
0621 07:48:06.023334 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0621 07:48:06.023370 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 07:48:06.024462 (+ 1092us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:48:06.024987 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:48:06.024999 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:48:06.025078 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:48:06.025554 (+ 476us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:48:06.025556 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:48:06.025590 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:48:06.025649 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:48:06.025929 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:48:06.025934 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:48:06.025938 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 07:48:06.026055 (+ 117us) client_negotiation.cc:240] Negotiation successful
0621 07:48:06.026248 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":112,"threads_started":1}
I20260621 07:48:06.028038 10822 heartbeater.cc:347] Connected to a master server at 127.0.192.254:34071
I20260621 07:48:06.028244 10822 heartbeater.cc:464] Registering TS with master...
I20260621 07:48:06.028659 10822 heartbeater.cc:511] Master 127.0.192.254:34071 requested a full tablet report, sending...
I20260621 07:48:06.029618 10503 ts_manager.cc:194] Registered new tserver with Master: bedf72935d114b66b62e957a90f3cb40 (127.0.192.194:38079)
I20260621 07:48:06.029875 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 10694
I20260621 07:48:06.030006 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-1/wal/instance
I20260621 07:48:06.030346 771 external_mini_cluster.cc:1428] Setting key 6755627eaa2d135df8081d89638607dc
I20260621 07:48:06.032037 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.195:0
--local_ip_for_outbound_sockets=127.0.192.195
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:34071
--builtin_ntp_servers=127.0.192.212:35023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260621 07:48:06.136987 10825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:48:06.137203 10825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:48:06.137269 10825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:48:06.141496 10825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:48:06.141541 10825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:48:06.141619 10825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.195
I20260621 07:48:06.146170 10825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:35023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:34071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.195
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:48:06.147078 10825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:48:06.148313 10825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:48:06.157331 10831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:48:06.157307 10830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:48:06.157459 10833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:48:06.157616 10825 server_base.cc:1034] running on GCE node
I20260621 07:48:06.158025 10825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:48:06.158591 10825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:48:06.159776 10825 hybrid_clock.cc:648] HybridClock initialized: now 1782028086159769 us; error 30 us; skew 500 ppm
I20260621 07:48:06.161450 10825 webserver.cc:460] Webserver started at http://127.0.192.195:39859/ using document root <none> and password file <none>
I20260621 07:48:06.161985 10825 fs_manager.cc:359] Metadata directory not provided
I20260621 07:48:06.162074 10825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:48:06.162289 10825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:48:06.164193 10825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-2/data/instance:
uuid: "08b900ed4a4b45baa32c9e8ba1b9de5f"
format_stamp: "Formatted at 2026-06-21 07:48:06 on dist-test-slave-lzx3"
server_key: "523096b7c6560b3b9ae62e637f40da70"
server_key_iv: "c3c6c11c7e21a11f0f8e3cfb0fe7908b"
server_key_version: "encryptionkey@0"
I20260621 07:48:06.164808 10825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-2/wal/instance:
uuid: "08b900ed4a4b45baa32c9e8ba1b9de5f"
format_stamp: "Formatted at 2026-06-21 07:48:06 on dist-test-slave-lzx3"
server_key: "523096b7c6560b3b9ae62e637f40da70"
server_key_iv: "c3c6c11c7e21a11f0f8e3cfb0fe7908b"
server_key_version: "encryptionkey@0"
I20260621 07:48:06.168463 10825 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260621 07:48:06.170723 10839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:48:06.171876 10825 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:48:06.172004 10825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-2/wal
uuid: "08b900ed4a4b45baa32c9e8ba1b9de5f"
format_stamp: "Formatted at 2026-06-21 07:48:06 on dist-test-slave-lzx3"
server_key: "523096b7c6560b3b9ae62e637f40da70"
server_key_iv: "c3c6c11c7e21a11f0f8e3cfb0fe7908b"
server_key_version: "encryptionkey@0"
I20260621 07:48:06.172116 10825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:48:06.176348 10825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:48:06.176968 10825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:48:06.177145 10825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:48:06.177662 10825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:48:06.178491 10825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:48:06.178560 10825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:48:06.178622 10825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:48:06.178664 10825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:48:06.188040 10825 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.195:34051
I20260621 07:48:06.188066 10952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.195:34051 every 8 connection(s)
I20260621 07:48:06.188901 10825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
I20260621 07:48:06.194459 10542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:06.190641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:44945 (local address 127.0.192.254:34071)
0621 07:48:06.190777 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:06.190781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:06.191718 (+ 937us) server_negotiation.cc:407] Connection header received
0621 07:48:06.192994 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:06.192997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:06.193033 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:06.193108 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:48:06.194139 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:48:06.194141 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:48:06.194143 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:48:06.194151 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:48:06.194168 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:48:06.194175 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:48:06.194240 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0621 07:48:06.194327 (+ 87us) server_negotiation.cc:299] Negotiation successful
0621 07:48:06.194361 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 07:48:06.195279 10955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:06.190998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:34071 (local address 127.0.192.195:44945)
0621 07:48:06.191597 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0621 07:48:06.191634 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 07:48:06.192816 (+ 1182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:48:06.193244 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:48:06.193250 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:48:06.193327 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:48:06.193915 (+ 588us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:48:06.193917 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:48:06.193953 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:48:06.194017 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:48:06.194228 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:48:06.194232 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:48:06.194235 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 07:48:06.194351 (+ 116us) client_negotiation.cc:240] Negotiation successful
0621 07:48:06.194550 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":97,"threads_started":1}
I20260621 07:48:06.196348 10953 heartbeater.cc:347] Connected to a master server at 127.0.192.254:34071
I20260621 07:48:06.196597 10953 heartbeater.cc:464] Registering TS with master...
I20260621 07:48:06.196965 10953 heartbeater.cc:511] Master 127.0.192.254:34071 requested a full tablet report, sending...
I20260621 07:48:06.197286 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 10825
I20260621 07:48:06.197398 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782028028522580-771-0/minicluster-data/ts-2/wal/instance
I20260621 07:48:06.197721 771 external_mini_cluster.cc:1428] Setting key 781abc9dec7c2111b0cc0449556af05a
I20260621 07:48:06.198076 10503 ts_manager.cc:194] Registered new tserver with Master: 08b900ed4a4b45baa32c9e8ba1b9de5f (127.0.192.195:34051)
I20260621 07:48:06.199741 771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260621 07:48:06.202353 10542 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:48:06.201580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47838 (local address 127.0.192.254:34071)
0621 07:48:06.201692 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:06.201696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:06.201854 (+ 158us) server_negotiation.cc:407] Connection header received
0621 07:48:06.201949 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:06.201951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:06.201978 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:06.202033 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:48:06.202205 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47838: BlockingRecv error: recv got EOF from 127.0.0.1:47838 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260621 07:48:06.202746 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 10563
I20260621 07:48:06.209095 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 10694
I20260621 07:48:06.214277 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 10825
I20260621 07:48:06.219267 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 10472
2026-06-21T07:48:06Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (769 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:48:06 dist-test-slave-lzx3 krb5kdc[10963](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:48:06 dist-test-slave-lzx3 krb5kdc[10963](info): set up 2 sockets
Jun 21 07:48:06 dist-test-slave-lzx3 krb5kdc[10963](info): commencing operation
krb5kdc: starting...
W20260621 07:48:07.205034 771 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/krb5kdc/test-user.keytab.
Jun 21 07:48:07 dist-test-slave-lzx3 krb5kdc[10963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:48:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:48:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.192.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
I20260621 07:48:07.307900 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:34121
--webserver_interface=127.0.192.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:33625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:34121
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:48:07.408615 10976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:48:07.408849 10976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:48:07.408883 10976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:48:07.412479 10976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:48:07.412521 10976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:48:07.412541 10976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:48:07.412611 10976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:48:07.412648 10976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:48:07.417179 10976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:33625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:34121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:34121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:48:07.417982 10976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:48:07.418807 10976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:48:07.426311 10981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:48:07.426375 10984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:48:07.426385 10976 server_base.cc:1034] running on GCE node
W20260621 07:48:07.426430 10982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:48:07.427029 10976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:48:07.427909 10976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:48:07.429057 10976 hybrid_clock.cc:648] HybridClock initialized: now 1782028087429046 us; error 44 us; skew 500 ppm
Jun 21 07:48:07 dist-test-slave-lzx3 krb5kdc[10963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:48:07.431443 10976 init.cc:377] Logged in from keytab as kudu/127.0.192.254@KRBTEST.COM (short username kudu)
I20260621 07:48:07.432224 10976 webserver.cc:460] Webserver started at http://127.0.192.254:38251/ using document root <none> and password file <none>
I20260621 07:48:07.432760 10976 fs_manager.cc:359] Metadata directory not provided
I20260621 07:48:07.432875 10976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:48:07.433102 10976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:48:07.434902 10976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "817382925e0642f6901ecb4d803dabe1"
format_stamp: "Formatted at 2026-06-21 07:48:07 on dist-test-slave-lzx3"
server_key: "da5b32e1653cd6d878839740ea41ea2d"
server_key_iv: "3f330196fb6fa3bf0bff7893349a2cd2"
server_key_version: "encryptionkey@0"
I20260621 07:48:07.435397 10976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "817382925e0642f6901ecb4d803dabe1"
format_stamp: "Formatted at 2026-06-21 07:48:07 on dist-test-slave-lzx3"
server_key: "da5b32e1653cd6d878839740ea41ea2d"
server_key_iv: "3f330196fb6fa3bf0bff7893349a2cd2"
server_key_version: "encryptionkey@0"
I20260621 07:48:07.438833 10976 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260621 07:48:07.441337 10991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:48:07.442386 10976 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260621 07:48:07.442513 10976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "817382925e0642f6901ecb4d803dabe1"
format_stamp: "Formatted at 2026-06-21 07:48:07 on dist-test-slave-lzx3"
server_key: "da5b32e1653cd6d878839740ea41ea2d"
server_key_iv: "3f330196fb6fa3bf0bff7893349a2cd2"
server_key_version: "encryptionkey@0"
I20260621 07:48:07.442615 10976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:48:07.459901 10976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:48:07.462615 10976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:48:07.462805 10976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:48:07.469182 10976 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:34121
I20260621 07:48:07.469463 11043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:34121 every 8 connection(s)
I20260621 07:48:07.470049 10976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:48:07.472385 11044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:48:07.472652 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 10976
I20260621 07:48:07.472774 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:48:07.473088 771 external_mini_cluster.cc:1428] Setting key f07118cb4f16fcf252a9bd6ac06bc007
I20260621 07:48:07.479326 11044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 817382925e0642f6901ecb4d803dabe1: Bootstrap starting.
I20260621 07:48:07.481653 11044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 817382925e0642f6901ecb4d803dabe1: Neither blocks nor log segments found. Creating new log.
Jun 21 07:48:07 dist-test-slave-lzx3 krb5kdc[10963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:48:07.482302 11044 log.cc:824] T 00000000000000000000000000000000 P 817382925e0642f6901ecb4d803dabe1: Log is configured to *not* fsync() on all Append() calls
I20260621 07:48:07.484000 11044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 817382925e0642f6901ecb4d803dabe1: No bootstrap required, opened a new log
I20260621 07:48:07.486312 11047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:07.474291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48318 (local address 127.0.192.254:34121)
0621 07:48:07.474628 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:07.474641 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:07.474672 (+ 31us) server_negotiation.cc:407] Connection header received
0621 07:48:07.475529 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:07.475550 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:07.475904 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:07.476242 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:48:07.477215 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:07.478305 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:07.479451 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:07.479874 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:07.479978 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:07.482280 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:48:07.482284 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:48:07.482295 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:48:07.482316 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:48:07.483947 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:48:07.484398 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:48:07.484403 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:48:07.484406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:48:07.484505 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:48:07.484784 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:48:07.484787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:48:07.484788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:48:07.485101 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:48:07.485281 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0621 07:48:07.485526 (+ 245us) server_negotiation.cc:299] Negotiation successful
0621 07:48:07.485751 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":111,"threads_started":1}
I20260621 07:48:07.487027 11044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 817382925e0642f6901ecb4d803dabe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "817382925e0642f6901ecb4d803dabe1" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34121 } }
I20260621 07:48:07.487219 11044 raft_consensus.cc:374] T 00000000000000000000000000000000 P 817382925e0642f6901ecb4d803dabe1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:48:07.487295 11044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 817382925e0642f6901ecb4d803dabe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 817382925e0642f6901ecb4d803dabe1, State: Initialized, Role: FOLLOWER
I20260621 07:48:07.487686 11044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 817382925e0642f6901ecb4d803dabe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "817382925e0642f6901ecb4d803dabe1" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34121 } }
I20260621 07:48:07.487825 11044 raft_consensus.cc:388] T 00000000000000000000000000000000 P 817382925e0642f6901ecb4d803dabe1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:48:07.487897 11044 raft_consensus.cc:482] T 00000000000000000000000000000000 P 817382925e0642f6901ecb4d803dabe1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:48:07.487994 11044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 817382925e0642f6901ecb4d803dabe1 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:48:07.488806 11044 raft_consensus.cc:504] T 00000000000000000000000000000000 P 817382925e0642f6901ecb4d803dabe1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "817382925e0642f6901ecb4d803dabe1" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34121 } }
I20260621 07:48:07.489053 11044 leader_election.cc:302] T 00000000000000000000000000000000 P 817382925e0642f6901ecb4d803dabe1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 817382925e0642f6901ecb4d803dabe1; no voters:
I20260621 07:48:07.489255 11044 leader_election.cc:288] T 00000000000000000000000000000000 P 817382925e0642f6901ecb4d803dabe1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:48:07.489362 11049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 817382925e0642f6901ecb4d803dabe1 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:48:07.489599 11049 raft_consensus.cc:686] T 00000000000000000000000000000000 P 817382925e0642f6901ecb4d803dabe1 [term 1 LEADER]: Becoming Leader. State: Replica: 817382925e0642f6901ecb4d803dabe1, State: Running, Role: LEADER
I20260621 07:48:07.489871 11049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 817382925e0642f6901ecb4d803dabe1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "817382925e0642f6901ecb4d803dabe1" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34121 } }
I20260621 07:48:07.490285 11044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 817382925e0642f6901ecb4d803dabe1 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:48:07.491276 11051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 817382925e0642f6901ecb4d803dabe1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 817382925e0642f6901ecb4d803dabe1. Latest consensus state: current_term: 1 leader_uuid: "817382925e0642f6901ecb4d803dabe1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "817382925e0642f6901ecb4d803dabe1" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34121 } } }
I20260621 07:48:07.491405 11051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 817382925e0642f6901ecb4d803dabe1 [sys.catalog]: This master's current role is: LEADER
I20260621 07:48:07.491676 11050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 817382925e0642f6901ecb4d803dabe1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "817382925e0642f6901ecb4d803dabe1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "817382925e0642f6901ecb4d803dabe1" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34121 } } }
I20260621 07:48:07.491745 11058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:48:07.491756 11050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 817382925e0642f6901ecb4d803dabe1 [sys.catalog]: This master's current role is: LEADER
I20260621 07:48:07.494633 11058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:48:07.499207 11058 catalog_manager.cc:1347] Generated new cluster ID: f2fb149884d746328641b8e5bd617d3e
I20260621 07:48:07.499262 11058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:48:07.530855 11058 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:48:07.531915 11058 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:48:07.544051 11058 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 817382925e0642f6901ecb4d803dabe1: Generated new TSK 0
I20260621 07:48:07.544693 11058 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.192.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab.
Entry for principal kudu/127.0.192.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab.
I20260621 07:48:07.568760 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.193:0
--local_ip_for_outbound_sockets=127.0.192.193
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:34121
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:33625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:48:07.673678 11070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:48:07.674055 11070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:48:07.674135 11070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:48:07.679466 11070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:48:07.679520 11070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:48:07.679620 11070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.193
I20260621 07:48:07.684335 11070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:33625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:34121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.193
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:48:07.685245 11070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:48:07.686118 11070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:48:07.694700 11075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:48:07.694722 11076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:48:07.694784 11070 server_base.cc:1034] running on GCE node
W20260621 07:48:07.694732 11078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:48:07.695273 11070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:48:07.695806 11070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:48:07.696961 11070 hybrid_clock.cc:648] HybridClock initialized: now 1782028087696937 us; error 36 us; skew 500 ppm
Jun 21 07:48:07 dist-test-slave-lzx3 krb5kdc[10963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:48:07.699442 11070 init.cc:377] Logged in from keytab as kudu/127.0.192.193@KRBTEST.COM (short username kudu)
I20260621 07:48:07.700153 11070 webserver.cc:460] Webserver started at http://127.0.192.193:36787/ using document root <none> and password file <none>
I20260621 07:48:07.700642 11070 fs_manager.cc:359] Metadata directory not provided
I20260621 07:48:07.700700 11070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:48:07.700839 11070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:48:07.702481 11070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-0/data/instance:
uuid: "78ed099da25f41f79080fdd37eebe019"
format_stamp: "Formatted at 2026-06-21 07:48:07 on dist-test-slave-lzx3"
server_key: "d4304e2f8a69e916d5a3074f3aab64ce"
server_key_iv: "591b54a61278b523033e29e7c9ace95e"
server_key_version: "encryptionkey@0"
I20260621 07:48:07.702940 11070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-0/wal/instance:
uuid: "78ed099da25f41f79080fdd37eebe019"
format_stamp: "Formatted at 2026-06-21 07:48:07 on dist-test-slave-lzx3"
server_key: "d4304e2f8a69e916d5a3074f3aab64ce"
server_key_iv: "591b54a61278b523033e29e7c9ace95e"
server_key_version: "encryptionkey@0"
I20260621 07:48:07.706197 11070 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 07:48:07.708432 11085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:48:07.709542 11070 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:48:07.709676 11070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-0/wal
uuid: "78ed099da25f41f79080fdd37eebe019"
format_stamp: "Formatted at 2026-06-21 07:48:07 on dist-test-slave-lzx3"
server_key: "d4304e2f8a69e916d5a3074f3aab64ce"
server_key_iv: "591b54a61278b523033e29e7c9ace95e"
server_key_version: "encryptionkey@0"
I20260621 07:48:07.709769 11070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:48:07.722556 11070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:48:07.725159 11070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:48:07.725337 11070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:48:07.725853 11070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:48:07.726509 11070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:48:07.726570 11070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:48:07.726627 11070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:48:07.726668 11070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:48:07.734197 11070 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.193:44701
I20260621 07:48:07.734211 11198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.193:44701 every 8 connection(s)
I20260621 07:48:07.735008 11070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
I20260621 07:48:07.743709 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 11070
I20260621 07:48:07.743836 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-0/wal/instance
I20260621 07:48:07.744196 771 external_mini_cluster.cc:1428] Setting key fe1a6405a043c33cff892d6510814ee4
Jun 21 07:48:07 dist-test-slave-lzx3 krb5kdc[10963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:48:07.748229 11047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:07.736635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:56015 (local address 127.0.192.254:34121)
0621 07:48:07.736769 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:07.736774 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:07.737612 (+ 838us) server_negotiation.cc:407] Connection header received
0621 07:48:07.739160 (+ 1548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:07.739162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:07.739235 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:07.739315 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:48:07.740352 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:07.740914 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:07.742039 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:07.742247 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:07.742353 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:07.744914 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:48:07.744918 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:48:07.744925 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:48:07.744954 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:48:07.746469 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:48:07.746923 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:48:07.746927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:48:07.746929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:48:07.747004 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:48:07.747302 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:48:07.747306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:48:07.747308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:48:07.747464 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:48:07.747560 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0621 07:48:07.747968 (+ 408us) server_negotiation.cc:299] Negotiation successful
0621 07:48:07.748081 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260621 07:48:07.748993 11201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:07.736998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:34121 (local address 127.0.192.193:56015)
0621 07:48:07.737462 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0621 07:48:07.737505 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 07:48:07.738872 (+ 1367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:48:07.739469 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:48:07.739475 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:48:07.739773 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:48:07.740217 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:07.740228 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:07.741068 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:07.741071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:07.741895 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:07.741904 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:07.742385 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:07.742389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:07.742558 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:07.743169 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:48:07.743184 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:48:07.744769 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:48:07.746576 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:48:07.746581 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:48:07.746593 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:48:07.746830 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:48:07.747088 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:48:07.747092 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:48:07.747094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:48:07.747218 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:48:07.747561 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:48:07.747567 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:48:07.747772 (+ 205us) client_negotiation.cc:769] Sending connection context
0621 07:48:07.747932 (+ 160us) client_negotiation.cc:240] Negotiation successful
0621 07:48:07.748191 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":93,"threads_started":1}
I20260621 07:48:07.750152 11199 heartbeater.cc:347] Connected to a master server at 127.0.192.254:34121
I20260621 07:48:07.750406 11199 heartbeater.cc:464] Registering TS with master...
I20260621 07:48:07.750927 11199 heartbeater.cc:511] Master 127.0.192.254:34121 requested a full tablet report, sending...
I20260621 07:48:07.752382 11008 ts_manager.cc:194] Registered new tserver with Master: 78ed099da25f41f79080fdd37eebe019 (127.0.192.193:44701)
WARNING: no policy specified for kudu/127.0.192.194@KRBTEST.COM; defaulting to no policy
I20260621 07:48:07.753568 11008 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.193@KRBTEST.COM'} at 127.0.192.193:56015
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab.
Entry for principal kudu/127.0.192.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab.
I20260621 07:48:07.769394 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.194:0
--local_ip_for_outbound_sockets=127.0.192.194
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:34121
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:33625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:48:07.867314 11204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:48:07.867517 11204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:48:07.867554 11204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:48:07.871824 11204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:48:07.871866 11204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:48:07.871915 11204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.194
I20260621 07:48:07.876463 11204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:33625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:34121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.194
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:48:07.877241 11204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:48:07.878033 11204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:48:07.886049 11212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:48:07.886049 11210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:48:07.886099 11209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:48:07.886610 11204 server_base.cc:1034] running on GCE node
I20260621 07:48:07.886880 11204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:48:07.887324 11204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:48:07.888469 11204 hybrid_clock.cc:648] HybridClock initialized: now 1782028087888449 us; error 31 us; skew 500 ppm
Jun 21 07:48:07 dist-test-slave-lzx3 krb5kdc[10963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:48:07.890872 11204 init.cc:377] Logged in from keytab as kudu/127.0.192.194@KRBTEST.COM (short username kudu)
I20260621 07:48:07.891731 11204 webserver.cc:460] Webserver started at http://127.0.192.194:36871/ using document root <none> and password file <none>
I20260621 07:48:07.892242 11204 fs_manager.cc:359] Metadata directory not provided
I20260621 07:48:07.892323 11204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:48:07.892524 11204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:48:07.894423 11204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-1/data/instance:
uuid: "441f815526c94001a9c9f73f19c51038"
format_stamp: "Formatted at 2026-06-21 07:48:07 on dist-test-slave-lzx3"
server_key: "e55692fb6938e46537be92a7027ee580"
server_key_iv: "16b1cc745d4f3a90d6a5bd99340d1b16"
server_key_version: "encryptionkey@0"
I20260621 07:48:07.894928 11204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-1/wal/instance:
uuid: "441f815526c94001a9c9f73f19c51038"
format_stamp: "Formatted at 2026-06-21 07:48:07 on dist-test-slave-lzx3"
server_key: "e55692fb6938e46537be92a7027ee580"
server_key_iv: "16b1cc745d4f3a90d6a5bd99340d1b16"
server_key_version: "encryptionkey@0"
I20260621 07:48:07.897919 11204 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260621 07:48:07.899925 11219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:48:07.900986 11204 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:48:07.901087 11204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-1/wal
uuid: "441f815526c94001a9c9f73f19c51038"
format_stamp: "Formatted at 2026-06-21 07:48:07 on dist-test-slave-lzx3"
server_key: "e55692fb6938e46537be92a7027ee580"
server_key_iv: "16b1cc745d4f3a90d6a5bd99340d1b16"
server_key_version: "encryptionkey@0"
I20260621 07:48:07.901175 11204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:48:07.922602 11204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:48:07.925227 11204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:48:07.925447 11204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:48:07.926011 11204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:48:07.926757 11204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:48:07.926801 11204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:48:07.926867 11204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:48:07.926911 11204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:48:07.934880 11204 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.194:34051
I20260621 07:48:07.934933 11332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.194:34051 every 8 connection(s)
I20260621 07:48:07.935622 11204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
Jun 21 07:48:07 dist-test-slave-lzx3 krb5kdc[10963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:48:07.944479 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 11204
I20260621 07:48:07.944624 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-1/wal/instance
I20260621 07:48:07.944989 771 external_mini_cluster.cc:1428] Setting key cf7cb8d14312ce4f1d94b88d2854cfaa
I20260621 07:48:07.947652 11047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:07.937089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:57625 (local address 127.0.192.254:34121)
0621 07:48:07.937204 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:07.937207 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:07.938275 (+ 1068us) server_negotiation.cc:407] Connection header received
0621 07:48:07.939060 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:07.939062 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:07.939109 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:07.939180 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:48:07.940198 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:07.940792 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:07.941820 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:07.941997 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:07.942068 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:07.944324 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:48:07.944330 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:48:07.944333 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:48:07.944361 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:48:07.945909 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:48:07.946360 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:48:07.946363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:48:07.946365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:48:07.946444 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:48:07.946760 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:48:07.946762 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:48:07.946764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:48:07.946909 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:48:07.947017 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0621 07:48:07.947415 (+ 398us) server_negotiation.cc:299] Negotiation successful
0621 07:48:07.947511 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260621 07:48:07.948261 11335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:07.937336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:34121 (local address 127.0.192.194:57625)
0621 07:48:07.938128 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0621 07:48:07.938167 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0621 07:48:07.938863 (+ 696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:48:07.939324 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:48:07.939330 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:48:07.939636 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:48:07.940064 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:07.940076 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:07.940907 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:07.940913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:07.941723 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:07.941732 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:07.942109 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:07.942111 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:07.942244 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:07.942733 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:48:07.942746 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:48:07.944122 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:48:07.946001 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:48:07.946005 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:48:07.946013 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:48:07.946274 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:48:07.946535 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:48:07.946538 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:48:07.946540 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:48:07.946659 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:48:07.947011 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:48:07.947016 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:48:07.947214 (+ 198us) client_negotiation.cc:769] Sending connection context
0621 07:48:07.947366 (+ 152us) client_negotiation.cc:240] Negotiation successful
0621 07:48:07.947626 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":500,"threads_started":1}
I20260621 07:48:07.949299 11333 heartbeater.cc:347] Connected to a master server at 127.0.192.254:34121
I20260621 07:48:07.949496 11333 heartbeater.cc:464] Registering TS with master...
I20260621 07:48:07.949947 11333 heartbeater.cc:511] Master 127.0.192.254:34121 requested a full tablet report, sending...
I20260621 07:48:07.950898 11008 ts_manager.cc:194] Registered new tserver with Master: 441f815526c94001a9c9f73f19c51038 (127.0.192.194:34051)
I20260621 07:48:07.951381 11008 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.194@KRBTEST.COM'} at 127.0.192.194:57625
WARNING: no policy specified for kudu/127.0.192.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab.
Entry for principal kudu/127.0.192.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab.
I20260621 07:48:07.971483 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.195:0
--local_ip_for_outbound_sockets=127.0.192.195
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:34121
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:33625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:48:08.066907 11338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:48:08.067139 11338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:48:08.067199 11338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:48:08.070679 11338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:48:08.070744 11338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:48:08.070814 11338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.195
I20260621 07:48:08.075304 11338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:33625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:34121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.195
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:48:08.076097 11338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:48:08.076960 11338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:48:08.085080 11343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:48:08.085181 11344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:48:08.085283 11338 server_base.cc:1034] running on GCE node
W20260621 07:48:08.085374 11346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:48:08.085728 11338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:48:08.086220 11338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:48:08.087389 11338 hybrid_clock.cc:648] HybridClock initialized: now 1782028088087377 us; error 35 us; skew 500 ppm
Jun 21 07:48:08 dist-test-slave-lzx3 krb5kdc[10963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028088, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:48:08.089689 11338 init.cc:377] Logged in from keytab as kudu/127.0.192.195@KRBTEST.COM (short username kudu)
I20260621 07:48:08.090502 11338 webserver.cc:460] Webserver started at http://127.0.192.195:33361/ using document root <none> and password file <none>
I20260621 07:48:08.090956 11338 fs_manager.cc:359] Metadata directory not provided
I20260621 07:48:08.091034 11338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:48:08.091202 11338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:48:08.093113 11338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-2/data/instance:
uuid: "a69b0d0fb205442aa809e4f1d494d21d"
format_stamp: "Formatted at 2026-06-21 07:48:08 on dist-test-slave-lzx3"
server_key: "b73cb7ed46b982cfc8656f5bceb3ca2f"
server_key_iv: "99bb78d530aac2455d2ec08d4a8c480e"
server_key_version: "encryptionkey@0"
I20260621 07:48:08.093640 11338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-2/wal/instance:
uuid: "a69b0d0fb205442aa809e4f1d494d21d"
format_stamp: "Formatted at 2026-06-21 07:48:08 on dist-test-slave-lzx3"
server_key: "b73cb7ed46b982cfc8656f5bceb3ca2f"
server_key_iv: "99bb78d530aac2455d2ec08d4a8c480e"
server_key_version: "encryptionkey@0"
I20260621 07:48:08.096767 11338 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:48:08.098904 11353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:48:08.099946 11338 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:48:08.100068 11338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-2/wal
uuid: "a69b0d0fb205442aa809e4f1d494d21d"
format_stamp: "Formatted at 2026-06-21 07:48:08 on dist-test-slave-lzx3"
server_key: "b73cb7ed46b982cfc8656f5bceb3ca2f"
server_key_iv: "99bb78d530aac2455d2ec08d4a8c480e"
server_key_version: "encryptionkey@0"
I20260621 07:48:08.100163 11338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:48:08.129717 11338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:48:08.132042 11338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:48:08.132257 11338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:48:08.132817 11338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:48:08.133576 11338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:48:08.133648 11338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:48:08.133713 11338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:48:08.133757 11338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:48:08.141696 11338 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.195:35293
I20260621 07:48:08.141728 11466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.195:35293 every 8 connection(s)
I20260621 07:48:08.142401 11338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
I20260621 07:48:08.146193 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 11338
I20260621 07:48:08.146324 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-2/wal/instance
I20260621 07:48:08.146641 771 external_mini_cluster.cc:1428] Setting key 9d169dc76c93a8e5e24f4571e499e005
Jun 21 07:48:08 dist-test-slave-lzx3 krb5kdc[10963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028088, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:48:08.154351 11047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:08.144022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:38793 (local address 127.0.192.254:34121)
0621 07:48:08.144194 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:08.144199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:08.145358 (+ 1159us) server_negotiation.cc:407] Connection header received
0621 07:48:08.146199 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:08.146202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:08.146277 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:08.146351 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:48:08.147561 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:08.148103 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:08.149259 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:08.149436 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:08.149507 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:08.151571 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:48:08.151574 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:48:08.151576 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:48:08.151593 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:48:08.152731 (+ 1138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:48:08.153178 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:48:08.153181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:48:08.153182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:48:08.153252 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:48:08.153537 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:48:08.153539 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:48:08.153540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:48:08.153685 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:48:08.153757 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0621 07:48:08.154179 (+ 422us) server_negotiation.cc:299] Negotiation successful
0621 07:48:08.154255 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260621 07:48:08.155043 11469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:08.144382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:34121 (local address 127.0.192.195:38793)
0621 07:48:08.145234 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0621 07:48:08.145272 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 07:48:08.146007 (+ 735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:48:08.146492 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:48:08.146501 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:48:08.146922 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:48:08.147427 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:08.147441 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:08.148215 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:08.148219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:08.149156 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:08.149165 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:08.149550 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:08.149553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:08.149674 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:08.150184 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:48:08.150199 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:48:08.151446 (+ 1247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:48:08.152819 (+ 1373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:48:08.152823 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:48:08.152831 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:48:08.153091 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:48:08.153330 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:48:08.153332 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:48:08.153334 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:48:08.153458 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:48:08.153769 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:48:08.153773 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:48:08.153965 (+ 192us) client_negotiation.cc:769] Sending connection context
0621 07:48:08.154131 (+ 166us) client_negotiation.cc:240] Negotiation successful
0621 07:48:08.154334 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":476,"threads_started":1}
I20260621 07:48:08.155974 11467 heartbeater.cc:347] Connected to a master server at 127.0.192.254:34121
I20260621 07:48:08.156169 11467 heartbeater.cc:464] Registering TS with master...
I20260621 07:48:08.156613 11467 heartbeater.cc:511] Master 127.0.192.254:34121 requested a full tablet report, sending...
I20260621 07:48:08.157464 11008 ts_manager.cc:194] Registered new tserver with Master: a69b0d0fb205442aa809e4f1d494d21d (127.0.192.195:35293)
I20260621 07:48:08.157959 11008 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.195@KRBTEST.COM'} at 127.0.192.195:38793
I20260621 07:48:08.160171 771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:48:08.167941 11047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:08.161856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48324 (local address 127.0.192.254:34121)
0621 07:48:08.161982 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:08.161986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:08.162117 (+ 131us) server_negotiation.cc:407] Connection header received
0621 07:48:08.162243 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:08.162246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:08.162278 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:08.162358 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:48:08.162918 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:08.163410 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:08.164263 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:08.164459 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:08.164583 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:08.165607 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:48:08.165611 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:48:08.165613 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:48:08.165632 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:48:08.166572 (+ 940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:48:08.166926 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:48:08.166930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:48:08.166932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:48:08.167002 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:48:08.167234 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:48:08.167238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:48:08.167240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:48:08.167407 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:48:08.167535 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0621 07:48:08.167721 (+ 186us) server_negotiation.cc:299] Negotiation successful
0621 07:48:08.167812 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 07:48:08.170121 11008 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:48:08.172868 11008 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:48:08.185437 11480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:08.180123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:34051 (local address 127.0.0.1:58292)
0621 07:48:08.180685 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0621 07:48:08.180696 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 07:48:08.180768 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:48:08.181417 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:48:08.181422 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:48:08.181443 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:48:08.181545 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:08.181553 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:08.182977 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:08.182981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:08.184029 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:08.184040 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:08.185013 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:08.185016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:08.185108 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:08.185146 (+ 38us) client_negotiation.cc:769] Sending connection context
0621 07:48:08.185236 (+ 90us) client_negotiation.cc:240] Negotiation successful
0621 07:48:08.185326 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":79,"threads_started":1}
I20260621 07:48:08.185647 11481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:08.180674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58292 (local address 127.0.192.194:34051)
0621 07:48:08.180961 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:08.180972 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:08.180995 (+ 23us) server_negotiation.cc:407] Connection header received
0621 07:48:08.181073 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:08.181079 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:08.181262 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:08.181689 (+ 427us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:48:08.181807 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:08.182823 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:08.184165 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:08.184902 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:08.185005 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:08.185129 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0621 07:48:08.185259 (+ 130us) server_negotiation.cc:299] Negotiation successful
0621 07:48:08.185345 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":111,"threads_started":1}
I20260621 07:48:08.185675 11482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:08.181099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:44701 (local address 127.0.0.1:46286)
0621 07:48:08.181304 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0621 07:48:08.181318 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:48:08.181391 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:48:08.182213 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:48:08.182216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:48:08.182230 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:48:08.182310 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:08.182315 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:08.183246 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:08.183249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:08.184268 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:08.184276 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:08.185330 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:08.185333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:08.185410 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:08.185429 (+ 19us) client_negotiation.cc:769] Sending connection context
0621 07:48:08.185494 (+ 65us) client_negotiation.cc:240] Negotiation successful
0621 07:48:08.185565 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":80,"threads_started":1}
I20260621 07:48:08.186023 11483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:08.181207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46286 (local address 127.0.192.193:44701)
0621 07:48:08.181912 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:08.181921 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:08.181944 (+ 23us) server_negotiation.cc:407] Connection header received
0621 07:48:08.181992 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:08.181995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:08.182101 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:08.182221 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:48:08.182392 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:08.183116 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:08.184396 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:08.185084 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:08.185173 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:08.185278 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0621 07:48:08.185813 (+ 535us) server_negotiation.cc:299] Negotiation successful
0621 07:48:08.185902 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":84,"threads_started":1}
I20260621 07:48:08.186519 11478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:08.179967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.195:35293 (local address 127.0.0.1:43506)
0621 07:48:08.180255 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0621 07:48:08.180295 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 07:48:08.180398 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:48:08.181661 (+ 1263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:48:08.181664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:48:08.181679 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:48:08.181770 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:08.181775 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:08.184184 (+ 2409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:08.184187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:08.184934 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:08.184943 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:08.186195 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:08.186198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:08.186272 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:08.186292 (+ 20us) client_negotiation.cc:769] Sending connection context
0621 07:48:08.186353 (+ 61us) client_negotiation.cc:240] Negotiation successful
0621 07:48:08.186410 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":77,"threads_started":1}
I20260621 07:48:08.186692 11479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:08.180061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43506 (local address 127.0.192.195:35293)
0621 07:48:08.180965 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:08.180977 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:08.180995 (+ 18us) server_negotiation.cc:407] Connection header received
0621 07:48:08.181054 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:08.181059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:08.181543 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:08.181674 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:48:08.181884 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:08.182823 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:08.185084 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:08.185934 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:08.186004 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:08.186106 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0621 07:48:08.186379 (+ 273us) server_negotiation.cc:299] Negotiation successful
0621 07:48:08.186448 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":813,"thread_start_us":122,"threads_started":1}
I20260621 07:48:08.188253 11267 tablet_service.cc:1467] Processing CreateTablet for tablet 9bc7a53374564c0a89d6666c336ca3fb (DEFAULT_TABLE table=test-table [id=4745ee4d261e4fc38a94dbeb2f666eb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:48:08.189138 11267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bc7a53374564c0a89d6666c336ca3fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:48:08.188855 11401 tablet_service.cc:1467] Processing CreateTablet for tablet 9bc7a53374564c0a89d6666c336ca3fb (DEFAULT_TABLE table=test-table [id=4745ee4d261e4fc38a94dbeb2f666eb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:48:08.189759 11401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bc7a53374564c0a89d6666c336ca3fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:48:08.190303 11133 tablet_service.cc:1467] Processing CreateTablet for tablet 9bc7a53374564c0a89d6666c336ca3fb (DEFAULT_TABLE table=test-table [id=4745ee4d261e4fc38a94dbeb2f666eb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:48:08.191179 11133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bc7a53374564c0a89d6666c336ca3fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:48:08.194867 11484 tablet_bootstrap.cc:492] T 9bc7a53374564c0a89d6666c336ca3fb P 441f815526c94001a9c9f73f19c51038: Bootstrap starting.
I20260621 07:48:08.196650 11485 tablet_bootstrap.cc:492] T 9bc7a53374564c0a89d6666c336ca3fb P a69b0d0fb205442aa809e4f1d494d21d: Bootstrap starting.
I20260621 07:48:08.197264 11486 tablet_bootstrap.cc:492] T 9bc7a53374564c0a89d6666c336ca3fb P 78ed099da25f41f79080fdd37eebe019: Bootstrap starting.
I20260621 07:48:08.198015 11484 tablet_bootstrap.cc:654] T 9bc7a53374564c0a89d6666c336ca3fb P 441f815526c94001a9c9f73f19c51038: Neither blocks nor log segments found. Creating new log.
I20260621 07:48:08.198649 11485 tablet_bootstrap.cc:654] T 9bc7a53374564c0a89d6666c336ca3fb P a69b0d0fb205442aa809e4f1d494d21d: Neither blocks nor log segments found. Creating new log.
I20260621 07:48:08.198832 11484 log.cc:824] T 9bc7a53374564c0a89d6666c336ca3fb P 441f815526c94001a9c9f73f19c51038: Log is configured to *not* fsync() on all Append() calls
I20260621 07:48:08.199397 11485 log.cc:824] T 9bc7a53374564c0a89d6666c336ca3fb P a69b0d0fb205442aa809e4f1d494d21d: Log is configured to *not* fsync() on all Append() calls
I20260621 07:48:08.199519 11486 tablet_bootstrap.cc:654] T 9bc7a53374564c0a89d6666c336ca3fb P 78ed099da25f41f79080fdd37eebe019: Neither blocks nor log segments found. Creating new log.
I20260621 07:48:08.200191 11486 log.cc:824] T 9bc7a53374564c0a89d6666c336ca3fb P 78ed099da25f41f79080fdd37eebe019: Log is configured to *not* fsync() on all Append() calls
I20260621 07:48:08.200677 11484 tablet_bootstrap.cc:492] T 9bc7a53374564c0a89d6666c336ca3fb P 441f815526c94001a9c9f73f19c51038: No bootstrap required, opened a new log
I20260621 07:48:08.200862 11484 ts_tablet_manager.cc:1397] T 9bc7a53374564c0a89d6666c336ca3fb P 441f815526c94001a9c9f73f19c51038: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260621 07:48:08.200954 11485 tablet_bootstrap.cc:492] T 9bc7a53374564c0a89d6666c336ca3fb P a69b0d0fb205442aa809e4f1d494d21d: No bootstrap required, opened a new log
I20260621 07:48:08.201105 11485 ts_tablet_manager.cc:1397] T 9bc7a53374564c0a89d6666c336ca3fb P a69b0d0fb205442aa809e4f1d494d21d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260621 07:48:08.201938 11486 tablet_bootstrap.cc:492] T 9bc7a53374564c0a89d6666c336ca3fb P 78ed099da25f41f79080fdd37eebe019: No bootstrap required, opened a new log
I20260621 07:48:08.202085 11486 ts_tablet_manager.cc:1397] T 9bc7a53374564c0a89d6666c336ca3fb P 78ed099da25f41f79080fdd37eebe019: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260621 07:48:08.204069 11485 raft_consensus.cc:348] T 9bc7a53374564c0a89d6666c336ca3fb P a69b0d0fb205442aa809e4f1d494d21d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a69b0d0fb205442aa809e4f1d494d21d" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 35293 } } peers { permanent_uuid: "441f815526c94001a9c9f73f19c51038" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 34051 } } peers { permanent_uuid: "78ed099da25f41f79080fdd37eebe019" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 44701 } }
I20260621 07:48:08.204381 11485 raft_consensus.cc:374] T 9bc7a53374564c0a89d6666c336ca3fb P a69b0d0fb205442aa809e4f1d494d21d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:48:08.204437 11485 raft_consensus.cc:729] T 9bc7a53374564c0a89d6666c336ca3fb P a69b0d0fb205442aa809e4f1d494d21d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a69b0d0fb205442aa809e4f1d494d21d, State: Initialized, Role: FOLLOWER
I20260621 07:48:08.204571 11484 raft_consensus.cc:348] T 9bc7a53374564c0a89d6666c336ca3fb P 441f815526c94001a9c9f73f19c51038 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a69b0d0fb205442aa809e4f1d494d21d" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 35293 } } peers { permanent_uuid: "441f815526c94001a9c9f73f19c51038" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 34051 } } peers { permanent_uuid: "78ed099da25f41f79080fdd37eebe019" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 44701 } }
I20260621 07:48:08.204816 11484 raft_consensus.cc:374] T 9bc7a53374564c0a89d6666c336ca3fb P 441f815526c94001a9c9f73f19c51038 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:48:08.204865 11484 raft_consensus.cc:729] T 9bc7a53374564c0a89d6666c336ca3fb P 441f815526c94001a9c9f73f19c51038 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 441f815526c94001a9c9f73f19c51038, State: Initialized, Role: FOLLOWER
I20260621 07:48:08.204883 11485 consensus_queue.cc:260] T 9bc7a53374564c0a89d6666c336ca3fb P a69b0d0fb205442aa809e4f1d494d21d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a69b0d0fb205442aa809e4f1d494d21d" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 35293 } } peers { permanent_uuid: "441f815526c94001a9c9f73f19c51038" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 34051 } } peers { permanent_uuid: "78ed099da25f41f79080fdd37eebe019" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 44701 } }
I20260621 07:48:08.205258 11484 consensus_queue.cc:260] T 9bc7a53374564c0a89d6666c336ca3fb P 441f815526c94001a9c9f73f19c51038 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a69b0d0fb205442aa809e4f1d494d21d" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 35293 } } peers { permanent_uuid: "441f815526c94001a9c9f73f19c51038" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 34051 } } peers { permanent_uuid: "78ed099da25f41f79080fdd37eebe019" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 44701 } }
I20260621 07:48:08.205420 11467 heartbeater.cc:503] Master 127.0.192.254:34121 was elected leader, sending a full tablet report...
I20260621 07:48:08.205423 11486 raft_consensus.cc:348] T 9bc7a53374564c0a89d6666c336ca3fb P 78ed099da25f41f79080fdd37eebe019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a69b0d0fb205442aa809e4f1d494d21d" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 35293 } } peers { permanent_uuid: "441f815526c94001a9c9f73f19c51038" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 34051 } } peers { permanent_uuid: "78ed099da25f41f79080fdd37eebe019" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 44701 } }
I20260621 07:48:08.205668 11486 raft_consensus.cc:374] T 9bc7a53374564c0a89d6666c336ca3fb P 78ed099da25f41f79080fdd37eebe019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:48:08.205714 11486 raft_consensus.cc:729] T 9bc7a53374564c0a89d6666c336ca3fb P 78ed099da25f41f79080fdd37eebe019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78ed099da25f41f79080fdd37eebe019, State: Initialized, Role: FOLLOWER
I20260621 07:48:08.206084 11484 ts_tablet_manager.cc:1428] T 9bc7a53374564c0a89d6666c336ca3fb P 441f815526c94001a9c9f73f19c51038: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260621 07:48:08.206048 11486 consensus_queue.cc:260] T 9bc7a53374564c0a89d6666c336ca3fb P 78ed099da25f41f79080fdd37eebe019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a69b0d0fb205442aa809e4f1d494d21d" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 35293 } } peers { permanent_uuid: "441f815526c94001a9c9f73f19c51038" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 34051 } } peers { permanent_uuid: "78ed099da25f41f79080fdd37eebe019" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 44701 } }
I20260621 07:48:08.206471 11333 heartbeater.cc:503] Master 127.0.192.254:34121 was elected leader, sending a full tablet report...
I20260621 07:48:08.206566 11485 ts_tablet_manager.cc:1428] T 9bc7a53374564c0a89d6666c336ca3fb P a69b0d0fb205442aa809e4f1d494d21d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260621 07:48:08.208461 11486 ts_tablet_manager.cc:1428] T 9bc7a53374564c0a89d6666c336ca3fb P 78ed099da25f41f79080fdd37eebe019: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260621 07:48:08.208787 11199 heartbeater.cc:503] Master 127.0.192.254:34121 was elected leader, sending a full tablet report...
W20260621 07:48:08.236011 11200 tablet.cc:2369] T 9bc7a53374564c0a89d6666c336ca3fb P 78ed099da25f41f79080fdd37eebe019: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:48:08.369539 11490 raft_consensus.cc:482] T 9bc7a53374564c0a89d6666c336ca3fb P a69b0d0fb205442aa809e4f1d494d21d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:48:08.369719 11490 raft_consensus.cc:504] T 9bc7a53374564c0a89d6666c336ca3fb P a69b0d0fb205442aa809e4f1d494d21d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a69b0d0fb205442aa809e4f1d494d21d" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 35293 } } peers { permanent_uuid: "441f815526c94001a9c9f73f19c51038" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 34051 } } peers { permanent_uuid: "78ed099da25f41f79080fdd37eebe019" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 44701 } }
I20260621 07:48:08.370925 11490 leader_election.cc:288] T 9bc7a53374564c0a89d6666c336ca3fb P a69b0d0fb205442aa809e4f1d494d21d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 441f815526c94001a9c9f73f19c51038 (127.0.192.194:34051), 78ed099da25f41f79080fdd37eebe019 (127.0.192.193:44701)
I20260621 07:48:08.374361 11481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:08.371307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:47307 (local address 127.0.192.194:34051)
0621 07:48:08.371446 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:08.371451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:08.371463 (+ 12us) server_negotiation.cc:407] Connection header received
0621 07:48:08.371505 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:08.371507 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:08.371545 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:08.371634 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:48:08.371909 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:08.372392 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:08.373418 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:08.373813 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:08.373942 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:08.373994 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0621 07:48:08.374171 (+ 177us) server_negotiation.cc:299] Negotiation successful
0621 07:48:08.374252 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 07:48:08.374361 11469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:08.371117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:34051 (local address 127.0.192.195:47307)
0621 07:48:08.371223 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0621 07:48:08.371241 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:48:08.371328 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:48:08.371643 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:48:08.371646 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:48:08.371662 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:48:08.371793 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:08.371798 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:08.372497 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:08.372502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:08.373291 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:08.373299 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:08.373930 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:08.373933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:08.374036 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:08.374066 (+ 30us) client_negotiation.cc:769] Sending connection context
0621 07:48:08.374156 (+ 90us) client_negotiation.cc:240] Negotiation successful
0621 07:48:08.374246 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"spinlock_wait_cycles":896}
I20260621 07:48:08.374578 11493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:08.371177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:44701 (local address 127.0.192.195:47967)
0621 07:48:08.371555 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0621 07:48:08.371566 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 07:48:08.371688 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:48:08.371954 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:48:08.371957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:48:08.371972 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:48:08.372097 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:08.372103 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:08.372783 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:08.372787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:08.373537 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:08.373546 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:08.374161 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:08.374165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:08.374283 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:08.374302 (+ 19us) client_negotiation.cc:769] Sending connection context
0621 07:48:08.374391 (+ 89us) client_negotiation.cc:240] Negotiation successful
0621 07:48:08.374450 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":117,"threads_started":1}
I20260621 07:48:08.374567 11483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:08.371293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:47967 (local address 127.0.192.193:44701)
0621 07:48:08.371416 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:08.371422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:08.371660 (+ 238us) server_negotiation.cc:407] Connection header received
0621 07:48:08.371808 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:08.371811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:08.371862 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:08.371949 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:48:08.372217 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:08.372676 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:08.373669 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:08.374041 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:08.374173 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:08.374247 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0621 07:48:08.374406 (+ 159us) server_negotiation.cc:299] Negotiation successful
0621 07:48:08.374469 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260621 07:48:08.375000 11287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bc7a53374564c0a89d6666c336ca3fb" candidate_uuid: "a69b0d0fb205442aa809e4f1d494d21d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "441f815526c94001a9c9f73f19c51038" is_pre_election: true
I20260621 07:48:08.375064 11153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bc7a53374564c0a89d6666c336ca3fb" candidate_uuid: "a69b0d0fb205442aa809e4f1d494d21d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78ed099da25f41f79080fdd37eebe019" is_pre_election: true
I20260621 07:48:08.375288 11287 raft_consensus.cc:2454] T 9bc7a53374564c0a89d6666c336ca3fb P 441f815526c94001a9c9f73f19c51038 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a69b0d0fb205442aa809e4f1d494d21d in term 0.
I20260621 07:48:08.375339 11153 raft_consensus.cc:2454] T 9bc7a53374564c0a89d6666c336ca3fb P 78ed099da25f41f79080fdd37eebe019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a69b0d0fb205442aa809e4f1d494d21d in term 0.
I20260621 07:48:08.375726 11354 leader_election.cc:302] T 9bc7a53374564c0a89d6666c336ca3fb P a69b0d0fb205442aa809e4f1d494d21d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 441f815526c94001a9c9f73f19c51038, a69b0d0fb205442aa809e4f1d494d21d; no voters:
I20260621 07:48:08.375969 11490 raft_consensus.cc:2781] T 9bc7a53374564c0a89d6666c336ca3fb P a69b0d0fb205442aa809e4f1d494d21d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:48:08.376045 11490 raft_consensus.cc:482] T 9bc7a53374564c0a89d6666c336ca3fb P a69b0d0fb205442aa809e4f1d494d21d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:48:08.376085 11490 raft_consensus.cc:3037] T 9bc7a53374564c0a89d6666c336ca3fb P a69b0d0fb205442aa809e4f1d494d21d [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:48:08.377027 11490 raft_consensus.cc:504] T 9bc7a53374564c0a89d6666c336ca3fb P a69b0d0fb205442aa809e4f1d494d21d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a69b0d0fb205442aa809e4f1d494d21d" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 35293 } } peers { permanent_uuid: "441f815526c94001a9c9f73f19c51038" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 34051 } } peers { permanent_uuid: "78ed099da25f41f79080fdd37eebe019" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 44701 } }
I20260621 07:48:08.377312 11490 leader_election.cc:288] T 9bc7a53374564c0a89d6666c336ca3fb P a69b0d0fb205442aa809e4f1d494d21d [CANDIDATE]: Term 1 election: Requested vote from peers 441f815526c94001a9c9f73f19c51038 (127.0.192.194:34051), 78ed099da25f41f79080fdd37eebe019 (127.0.192.193:44701)
I20260621 07:48:08.377583 11287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bc7a53374564c0a89d6666c336ca3fb" candidate_uuid: "a69b0d0fb205442aa809e4f1d494d21d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "441f815526c94001a9c9f73f19c51038"
I20260621 07:48:08.377687 11153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bc7a53374564c0a89d6666c336ca3fb" candidate_uuid: "a69b0d0fb205442aa809e4f1d494d21d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78ed099da25f41f79080fdd37eebe019"
I20260621 07:48:08.377707 11287 raft_consensus.cc:3037] T 9bc7a53374564c0a89d6666c336ca3fb P 441f815526c94001a9c9f73f19c51038 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:48:08.377803 11153 raft_consensus.cc:3037] T 9bc7a53374564c0a89d6666c336ca3fb P 78ed099da25f41f79080fdd37eebe019 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:48:08.378949 11153 raft_consensus.cc:2454] T 9bc7a53374564c0a89d6666c336ca3fb P 78ed099da25f41f79080fdd37eebe019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a69b0d0fb205442aa809e4f1d494d21d in term 1.
I20260621 07:48:08.378949 11287 raft_consensus.cc:2454] T 9bc7a53374564c0a89d6666c336ca3fb P 441f815526c94001a9c9f73f19c51038 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a69b0d0fb205442aa809e4f1d494d21d in term 1.
I20260621 07:48:08.379217 11354 leader_election.cc:302] T 9bc7a53374564c0a89d6666c336ca3fb P a69b0d0fb205442aa809e4f1d494d21d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 441f815526c94001a9c9f73f19c51038, a69b0d0fb205442aa809e4f1d494d21d; no voters:
I20260621 07:48:08.379393 11490 raft_consensus.cc:2781] T 9bc7a53374564c0a89d6666c336ca3fb P a69b0d0fb205442aa809e4f1d494d21d [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:48:08.379578 11490 raft_consensus.cc:686] T 9bc7a53374564c0a89d6666c336ca3fb P a69b0d0fb205442aa809e4f1d494d21d [term 1 LEADER]: Becoming Leader. State: Replica: a69b0d0fb205442aa809e4f1d494d21d, State: Running, Role: LEADER
I20260621 07:48:08.379858 11490 consensus_queue.cc:237] T 9bc7a53374564c0a89d6666c336ca3fb P a69b0d0fb205442aa809e4f1d494d21d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a69b0d0fb205442aa809e4f1d494d21d" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 35293 } } peers { permanent_uuid: "441f815526c94001a9c9f73f19c51038" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 34051 } } peers { permanent_uuid: "78ed099da25f41f79080fdd37eebe019" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 44701 } }
I20260621 07:48:08.382670 11008 catalog_manager.cc:5495] T 9bc7a53374564c0a89d6666c336ca3fb P a69b0d0fb205442aa809e4f1d494d21d reported cstate change: term changed from 0 to 1, leader changed from <none> to a69b0d0fb205442aa809e4f1d494d21d (127.0.192.195). New cstate: current_term: 1 leader_uuid: "a69b0d0fb205442aa809e4f1d494d21d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a69b0d0fb205442aa809e4f1d494d21d" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 35293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441f815526c94001a9c9f73f19c51038" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 34051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78ed099da25f41f79080fdd37eebe019" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 44701 } health_report { overall_health: UNKNOWN } } }
W20260621 07:48:08.393224 11468 tablet.cc:2369] T 9bc7a53374564c0a89d6666c336ca3fb P a69b0d0fb205442aa809e4f1d494d21d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:48:08.415307 11479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:08.412347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43508 (local address 127.0.192.195:35293)
0621 07:48:08.412456 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:08.412460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:08.412503 (+ 43us) server_negotiation.cc:407] Connection header received
0621 07:48:08.412600 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:08.412603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:08.412654 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:08.412732 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:48:08.412927 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:08.413471 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:08.414281 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:08.414453 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:08.414523 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:08.414732 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:48:08.415058 (+ 326us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:48:08.415135 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0621 07:48:08.415186 (+ 51us) server_negotiation.cc:299] Negotiation successful
0621 07:48:08.415226 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260621 07:48:08.420421 11287 raft_consensus.cc:1261] T 9bc7a53374564c0a89d6666c336ca3fb P 441f815526c94001a9c9f73f19c51038 [term 1 FOLLOWER]: Refusing update from remote peer a69b0d0fb205442aa809e4f1d494d21d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:48:08.420471 11153 raft_consensus.cc:1261] T 9bc7a53374564c0a89d6666c336ca3fb P 78ed099da25f41f79080fdd37eebe019 [term 1 FOLLOWER]: Refusing update from remote peer a69b0d0fb205442aa809e4f1d494d21d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:48:08.421129 11494 consensus_queue.cc:1035] T 9bc7a53374564c0a89d6666c336ca3fb P a69b0d0fb205442aa809e4f1d494d21d [LEADER]: Connected to new peer: Peer: permanent_uuid: "441f815526c94001a9c9f73f19c51038" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 34051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:48:08.421290 11490 consensus_queue.cc:1035] T 9bc7a53374564c0a89d6666c336ca3fb P a69b0d0fb205442aa809e4f1d494d21d [LEADER]: Connected to new peer: Peer: permanent_uuid: "78ed099da25f41f79080fdd37eebe019" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 44701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:48:08.428673 11498 mvcc.cc:205] Tried to move back new op lower bound from 7299187050163224576 to 7299187050006011904. Current Snapshot: MvccSnapshot[applied={T|T < 7299187050163224576}]
I20260621 07:48:08.430678 11499 mvcc.cc:205] Tried to move back new op lower bound from 7299187050163224576 to 7299187050006011904. Current Snapshot: MvccSnapshot[applied={T|T < 7299187050163224576 or (T in {7299187050163224576})}]
I20260621 07:48:08.435328 11006 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:48:08.435478 11006 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:48:08.437677 11006 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 817382925e0642f6901ecb4d803dabe1: Sending DeleteTablet for 3 replicas of tablet 9bc7a53374564c0a89d6666c336ca3fb
I20260621 07:48:08.438318 11401 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bc7a53374564c0a89d6666c336ca3fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:48:08 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:43506
I20260621 07:48:08.438385 11267 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bc7a53374564c0a89d6666c336ca3fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:48:08 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:58292
I20260621 07:48:08.438558 11133 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bc7a53374564c0a89d6666c336ca3fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:48:08 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:46286
I20260621 07:48:08.438813 11506 tablet_replica.cc:330] T 9bc7a53374564c0a89d6666c336ca3fb P a69b0d0fb205442aa809e4f1d494d21d: stopping tablet replica
I20260621 07:48:08.439015 11508 tablet_replica.cc:330] T 9bc7a53374564c0a89d6666c336ca3fb P 78ed099da25f41f79080fdd37eebe019: stopping tablet replica
I20260621 07:48:08.439108 11506 raft_consensus.cc:2229] T 9bc7a53374564c0a89d6666c336ca3fb P a69b0d0fb205442aa809e4f1d494d21d [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:48:08.439347 11508 raft_consensus.cc:2229] T 9bc7a53374564c0a89d6666c336ca3fb P 78ed099da25f41f79080fdd37eebe019 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:48:08.439388 11506 raft_consensus.cc:2258] T 9bc7a53374564c0a89d6666c336ca3fb P a69b0d0fb205442aa809e4f1d494d21d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:48:08.439517 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 11070
I20260621 07:48:08.439672 11508 raft_consensus.cc:2258] T 9bc7a53374564c0a89d6666c336ca3fb P 78ed099da25f41f79080fdd37eebe019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:48:08.441206 11506 ts_tablet_manager.cc:1905] T 9bc7a53374564c0a89d6666c336ca3fb P a69b0d0fb205442aa809e4f1d494d21d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:48:08.444419 11506 ts_tablet_manager.cc:1918] T 9bc7a53374564c0a89d6666c336ca3fb P a69b0d0fb205442aa809e4f1d494d21d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:48:08.444530 11506 log.cc:1196] T 9bc7a53374564c0a89d6666c336ca3fb P a69b0d0fb205442aa809e4f1d494d21d: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782028028522580-771-0/minicluster-data/ts-2/wal/wals/9bc7a53374564c0a89d6666c336ca3fb
I20260621 07:48:08.444890 11506 ts_tablet_manager.cc:1939] T 9bc7a53374564c0a89d6666c336ca3fb P a69b0d0fb205442aa809e4f1d494d21d: Deleting consensus metadata
I20260621 07:48:08.445961 10995 catalog_manager.cc:4841] TS a69b0d0fb205442aa809e4f1d494d21d (127.0.192.195:35293): tablet 9bc7a53374564c0a89d6666c336ca3fb (table test-table [id=4745ee4d261e4fc38a94dbeb2f666eb8]) successfully deleted
W20260621 07:48:08.447072 10993 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.192.193:44701 (error 108)
I20260621 07:48:08.447288 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 11204
I20260621 07:48:08.448458 11478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:08.448165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:44701 (local address 127.0.0.1:46288)
0621 07:48:08.448315 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0621 07:48:08.448378 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.193:44701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260621 07:48:08.448607 10993 catalog_manager.cc:4584] TS 78ed099da25f41f79080fdd37eebe019 (127.0.192.193:44701): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9bc7a53374564c0a89d6666c336ca3fb: Network error: Client connection negotiation failed: client connection to 127.0.192.193:44701: connect: Connection refused (error 111)
I20260621 07:48:08.453842 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 11338
I20260621 07:48:08.453970 11478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:08.453739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:34051 (local address 127.0.0.1:58302)
0621 07:48:08.453852 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0621 07:48:08.453903 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.194:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260621 07:48:08.457907 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 10976
2026-06-21T07:48:08Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2237 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:48:08 dist-test-slave-lzx3 krb5kdc[11512](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:48:08 dist-test-slave-lzx3 krb5kdc[11512](info): set up 2 sockets
Jun 21 07:48:08 dist-test-slave-lzx3 krb5kdc[11512](info): commencing operation
krb5kdc: starting...
W20260621 07:48:09.408953 771 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/krb5kdc/test-user.keytab.
Jun 21 07:48:09 dist-test-slave-lzx3 krb5kdc[11512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:48:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:48:09Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.192.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.192.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.192.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/krb5kdc/rangeradmin_127.0.192.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.192.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/krb5kdc/rangeradmin_127.0.192.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.192.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.192.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.192.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/krb5kdc/rangerlookup_127.0.192.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.192.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/krb5kdc/rangerlookup_127.0.192.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.192.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.192.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.192.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/krb5kdc/HTTP_127.0.192.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.192.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/krb5kdc/HTTP_127.0.192.212@KRBTEST.COM.keytab.
I20260621 07:48:09.554879 771 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskeKA8lE/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/postgres -l logfile start
2026-06-21 07:48:11.756 UTC [11542] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-21 07:48:11.756 UTC [11542] LOG: listening on IPv4 address "127.0.192.212", port 37201
2026-06-21 07:48:11.760 UTC [11542] LOG: listening on Unix socket "/tmp/.s.PGSQL.37201"
2026-06-21 07:48:11.779 UTC [11545] LOG: database system was shut down at 2026-06-21 07:48:10 UTC
2026-06-21 07:48:11.782 UTC [11542] LOG: database system is ready to accept connections
I20260621 07:48:12.628974 771 mini_postgres.cc:96] Postgres bound to 37201
2026-06-21 07:48:12.634 UTC [11553] FATAL: database "slave" does not exist
127.0.192.212:37201 - accepting connections
I20260621 07:48:12.635471 771 mini_ranger.cc:161] Starting Ranger...
I20260621 07:48:12.649132 771 mini_ranger.cc:84] Created miniranger Postgres user
I20260621 07:48:13.084404 771 mini_ranger.cc:87] Created ranger Postgres database
I20260621 07:48:13.084510 771 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ranger-admin
2026-06-21 07:48:13,339 [I] DB FLAVOR :POSTGRES
2026-06-21 07:48:13,339 [I] --------- Verifying Ranger DB connection ---------
2026-06-21 07:48:13,339 [I] Checking connection..
2026-06-21 07:48:13,339 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:37201/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-21 07:48:13,573 [I] Checking connection passed.
2026-06-21 07:48:13,573 [I] --------- Verifying version history table ---------
2026-06-21 07:48:13,573 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:37201/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 07:48:13,798 [I] Table x_db_version_h does not exist in database ranger
2026-06-21 07:48:13,799 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:37201/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 07:48:14,017 [I] Table x_db_version_h does not exist in database ranger
2026-06-21 07:48:14,018 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-21 07:48:14,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:37201/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-21 07:48:14.235 UTC [11617] WARNING: there is no transaction in progress
2026-06-21 07:48:14,240 [I] create_dbversion_catalog.sql file imported successfully
2026-06-21 07:48:14,241 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:37201/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 07:48:14,442 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-21 07:48:14,443 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-21 07:48:14,443 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:37201/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-21 07:48:14,669 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:37201/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-21 07:48:14,875 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:37201/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-lzx3.c.gcp-upstream.internal','N') ;"
2026-06-21 07:48:15,109 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:37201/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-21 07:48:15,311 [I] Table x_portal_user does not exist in database ranger
2026-06-21 07:48:15,311 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:37201/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-21 07:48:15,521 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-21 07:48:15,521 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:37201/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-21 07:48:15,738 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:37201/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-21 07:48:15,940 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-21 07:48:15,941 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:37201/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-21 07:48:16.783 UTC [11725] WARNING: there is no transaction in progress
2026-06-21 07:48:16.797 UTC [11725] WARNING: there is no transaction in progress
2026-06-21 07:48:16.811 UTC [11725] WARNING: there is no transaction in progress
2026-06-21 07:48:16.824 UTC [11725] WARNING: there is no transaction in progress
2026-06-21 07:48:16.839 UTC [11725] WARNING: there is no transaction in progress
2026-06-21 07:48:16.854 UTC [11725] WARNING: there is no transaction in progress
2026-06-21 07:48:16.909 UTC [11725] WARNING: there is no transaction in progress
2026-06-21 07:48:16.916 UTC [11725] WARNING: there is no transaction in progress
2026-06-21 07:48:16.930 UTC [11725] WARNING: there is no transaction in progress
2026-06-21 07:48:16.940 UTC [11725] WARNING: there is no transaction in progress
2026-06-21 07:48:16.950 UTC [11725] WARNING: there is no transaction in progress
2026-06-21 07:48:16.963 UTC [11725] WARNING: there is no transaction in progress
2026-06-21 07:48:16.973 UTC [11725] WARNING: there is no transaction in progress
2026-06-21 07:48:16.979 UTC [11725] WARNING: there is no transaction in progress
2026-06-21 07:48:16.986 UTC [11725] WARNING: there is no transaction in progress
2026-06-21 07:48:16.992 UTC [11725] WARNING: there is no transaction in progress
2026-06-21 07:48:17,561 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-21 07:48:17,562 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:37201/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-21 07:48:17,790 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-21 07:48:17,790 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:37201/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-21 07:48:18,024 [I] Table x_portal_user already exists in database 'ranger'
2026-06-21 07:48:18,025 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:37201/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-21 07:48:18,261 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-21 07:48:18,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:37201/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-21 07:48:18,481 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:37201/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-21 07:48:18,712 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:37201/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-lzx3.c.gcp-upstream.internal';"
2026-06-21 07:48:18,929 [I] CORE_DB_SCHEMA import status has been updated
2026-06-21 07:48:18,929 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:37201/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-21 07:48:19,138 [I] DB_PATCHES have already been applied
I20260621 07:48:19.144009 771 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ranger-admin:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskeKA8lE/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260621 07:48:19.144069 771 mini_ranger.cc:193] Using host: 127.0.192.212
I20260621 07:48:19.147346 771 mini_ranger.cc:227] Ranger admin URL: http://127.0.192.212:40661
Jun 21, 2026 7:48:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 21, 2026 7:48:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 21, 2026 7:48:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ranger-admin/ews/webapp
Jun 21, 2026 7:48:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 21, 2026 7:48:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 21, 2026 7:48:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ranger-admin/ews/webapp].
Jun 21, 2026 7:48:20 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260621 07:48:21.236627 771 mini_ranger.cc:160] Time spent starting Ranger: real 8.601s user 0.003s sys 0.008s
Jun 21, 2026 7:48:39 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 21, 2026 7:48:39 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 21, 2026 7:48:39 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 21, 2026 7:48:39 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 21, 2026 7:48:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 21, 2026 7:48:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 21, 2026 7:48:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 21, 2026 7:48:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 21, 2026 7:48:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 21, 2026 7:48:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 21, 2026 7:48:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 21, 2026 7:48:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 21, 2026 7:48:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 21, 2026 7:48:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 21, 2026 7:48:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 21, 2026 7:48:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 21, 2026 7:48:39 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 21, 2026 7:48:39 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260621 07:48:40.440960 771 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.192.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.192.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.192.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/krb5kdc/rangerkms_127.0.192.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.192.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/krb5kdc/rangerkms_127.0.192.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.192.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.192.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.192.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/krb5kdc/HTTP_127.0.192.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.192.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/krb5kdc/HTTP_127.0.192.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 21 07:48:40 dist-test-slave-lzx3 krb5kdc[11512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028120, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260621 07:48:40.519191 771 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260621 07:48:40.528180 771 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260621 07:48:41.296147 771 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260621 07:48:41.296241 771 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ranger-kms
I20260621 07:48:41.296348 771 mini_ranger_kms.cc:227] Using postgres at 127.0.192.212:37201
2026-06-21 07:48:41,467 [I] DB FLAVOR :POSTGRES
2026-06-21 07:48:41,468 [I] --------- Verifying Ranger DB connection ---------
2026-06-21 07:48:41,468 [I] Checking connection
2026-06-21 07:48:41,468 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:37201/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-21 07:48:41,699 [I] connection success
2026-06-21 07:48:41,700 [I] --------- Verifying Ranger DB tables ---------
2026-06-21 07:48:41,700 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-21 07:48:41,700 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:37201/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-21 07:48:41,906 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-21 07:48:41,906 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-21 07:48:41,906 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-21 07:48:41,907 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:37201/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-21 07:48:42,129 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260621 07:48:42.525395 771 mini_ranger_kms.cc:310] Created kms service
I20260621 07:48:42.680787 771 mini_ranger_kms.cc:326] Created kudu user
I20260621 07:48:42.720710 771 mini_ranger_kms.cc:343] Created rangerkms user
Jun 21 07:48:42 dist-test-slave-lzx3 krb5kdc[11512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028120, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.192.212@KRBTEST.COM
I20260621 07:48:43.087365 771 mini_ranger_kms.cc:384] Added ranger policy
I20260621 07:48:43.087687 771 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ranger-kms:/tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260621 07:48:43.087723 771 mini_ranger_kms.cc:240] Using host: 127.0.192.212
I20260621 07:48:43.091384 771 mini_ranger_kms.cc:276] Ranger KMS PID: 11906
I20260621 07:48:43.091488 771 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.192.212:54807
Jun 21, 2026 7:48:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 21, 2026 7:48:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 21, 2026 7:48:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 21, 2026 7:48:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 21, 2026 7:48:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ranger-kms/ews/webapp].
Jun 21, 2026 7:48:44 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260621 07:48:45.156806 771 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.638s user 0.004s sys 0.152s
I20260621 07:48:45.156924 771 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260621 07:48:45.157016 771 mini_ranger_kms.cc:401] 127.0.192.212:54807/kms/v1/keys
Jun 21 07:48:45 dist-test-slave-lzx3 krb5kdc[11512](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782028125, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.192.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 21 07:48:45 dist-test-slave-lzx3 krb5kdc[11512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028125, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.192.212@KRBTEST.COM for HTTP/127.0.192.212@KRBTEST.COM
Jun 21 07:48:45 dist-test-slave-lzx3 krb5kdc[11512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028125, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.192.212@KRBTEST.COM for HTTP/127.0.192.212@KRBTEST.COM
Jun 21 07:48:47 dist-test-slave-lzx3 krb5kdc[11512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.192.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
I20260621 07:48:48.003305 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:44841
--webserver_interface=127.0.192.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:40489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:44841
--ranger_config_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.192.212:54807/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:48:48.117688 11957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:48:48.117914 11957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:48:48.117949 11957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:48:48.121479 11957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:48:48.121539 11957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:48:48.121559 11957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:48:48.121575 11957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:48:48.121590 11957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:48:48.126607 11957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:40489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.192.212:54807/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:44841
--ranger_config_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:44841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:48:48.127602 11957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:48:48.128542 11957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:48:48.136082 11962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:48:48.136101 11963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:48:48.136351 11965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:48:48.136799 11957 server_base.cc:1034] running on GCE node
I20260621 07:48:48.137329 11957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:48:48.138356 11957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:48:48.139575 11957 hybrid_clock.cc:648] HybridClock initialized: now 1782028128139548 us; error 36 us; skew 500 ppm
Jun 21 07:48:48 dist-test-slave-lzx3 krb5kdc[11512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:48:48.142472 11957 init.cc:377] Logged in from keytab as kudu/127.0.192.254@KRBTEST.COM (short username kudu)
I20260621 07:48:48.143447 11957 webserver.cc:460] Webserver started at http://127.0.192.254:41833/ using document root <none> and password file <none>
I20260621 07:48:48.143929 11957 fs_manager.cc:359] Metadata directory not provided
I20260621 07:48:48.144004 11957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:48:48.144212 11957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 07:48:48 dist-test-slave-lzx3 krb5kdc[11512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for HTTP/127.0.192.212@KRBTEST.COM
I20260621 07:48:48.188601 11957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "d2ad6971df1e4cc688fb208e70d83a14"
format_stamp: "Formatted at 2026-06-21 07:48:48 on dist-test-slave-lzx3"
server_key: "6ee4d508aa477bea43b435b1b701f388"
server_key_iv: "147924444b79783a1184f8f9cf8d48b8"
server_key_version: "kuduclusterkey@0"
I20260621 07:48:48.189215 11957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "d2ad6971df1e4cc688fb208e70d83a14"
format_stamp: "Formatted at 2026-06-21 07:48:48 on dist-test-slave-lzx3"
server_key: "6ee4d508aa477bea43b435b1b701f388"
server_key_iv: "147924444b79783a1184f8f9cf8d48b8"
server_key_version: "kuduclusterkey@0"
I20260621 07:48:48.192931 11957 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260621 07:48:48.205371 11972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:48:48.206621 11957 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:48:48.206753 11957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "d2ad6971df1e4cc688fb208e70d83a14"
format_stamp: "Formatted at 2026-06-21 07:48:48 on dist-test-slave-lzx3"
server_key: "6ee4d508aa477bea43b435b1b701f388"
server_key_iv: "147924444b79783a1184f8f9cf8d48b8"
server_key_version: "kuduclusterkey@0"
I20260621 07:48:48.206876 11957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:48:48.230762 11957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:48:48.233675 11957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:48:48.233865 11957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:48:48.242383 11957 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:44841
I20260621 07:48:48.242410 12024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:44841 every 8 connection(s)
I20260621 07:48:48.243386 11957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:48:48.246461 12025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:48:48.249938 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 11957
I20260621 07:48:48.250067 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:48:48.250356 771 external_mini_cluster.cc:1428] Setting key 44ceff22806d51c0699e1f9b9d2bd9a2
I20260621 07:48:48.253091 12025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2ad6971df1e4cc688fb208e70d83a14: Bootstrap starting.
I20260621 07:48:48.255474 12025 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2ad6971df1e4cc688fb208e70d83a14: Neither blocks nor log segments found. Creating new log.
I20260621 07:48:48.256297 12025 log.cc:824] T 00000000000000000000000000000000 P d2ad6971df1e4cc688fb208e70d83a14: Log is configured to *not* fsync() on all Append() calls
I20260621 07:48:48.258255 12025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2ad6971df1e4cc688fb208e70d83a14: No bootstrap required, opened a new log
Jun 21 07:48:48 dist-test-slave-lzx3 krb5kdc[11512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:48:48.262252 12025 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2ad6971df1e4cc688fb208e70d83a14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ad6971df1e4cc688fb208e70d83a14" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 44841 } }
I20260621 07:48:48.262526 12025 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2ad6971df1e4cc688fb208e70d83a14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:48:48.262598 12025 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2ad6971df1e4cc688fb208e70d83a14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2ad6971df1e4cc688fb208e70d83a14, State: Initialized, Role: FOLLOWER
I20260621 07:48:48.263059 12025 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2ad6971df1e4cc688fb208e70d83a14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ad6971df1e4cc688fb208e70d83a14" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 44841 } }
I20260621 07:48:48.263206 12025 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2ad6971df1e4cc688fb208e70d83a14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:48:48.263284 12025 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2ad6971df1e4cc688fb208e70d83a14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:48:48.263377 12025 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2ad6971df1e4cc688fb208e70d83a14 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:48:48.264249 12025 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2ad6971df1e4cc688fb208e70d83a14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ad6971df1e4cc688fb208e70d83a14" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 44841 } }
I20260621 07:48:48.264648 12025 leader_election.cc:302] T 00000000000000000000000000000000 P d2ad6971df1e4cc688fb208e70d83a14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2ad6971df1e4cc688fb208e70d83a14; no voters:
I20260621 07:48:48.264990 12025 leader_election.cc:288] T 00000000000000000000000000000000 P d2ad6971df1e4cc688fb208e70d83a14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:48:48.265101 12030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2ad6971df1e4cc688fb208e70d83a14 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:48:48.265167 12028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:48.251734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44582 (local address 127.0.192.254:44841)
0621 07:48:48.252110 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:48.252146 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:48.252178 (+ 32us) server_negotiation.cc:407] Connection header received
0621 07:48:48.252775 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:48.252792 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:48.253092 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:48.253581 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:48:48.254243 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:48.255065 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:48.255890 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:48.256292 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:48.256451 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:48.259675 (+ 3224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:48:48.259684 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:48:48.259701 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:48:48.259729 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:48:48.261644 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:48:48.262145 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:48:48.262150 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:48:48.262155 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:48:48.262298 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:48:48.262623 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:48:48.262627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:48:48.262628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:48:48.263114 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:48:48.263308 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0621 07:48:48.264023 (+ 715us) server_negotiation.cc:299] Negotiation successful
0621 07:48:48.264255 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":134,"threads_started":1}
I20260621 07:48:48.265303 12030 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2ad6971df1e4cc688fb208e70d83a14 [term 1 LEADER]: Becoming Leader. State: Replica: d2ad6971df1e4cc688fb208e70d83a14, State: Running, Role: LEADER
I20260621 07:48:48.266175 12025 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2ad6971df1e4cc688fb208e70d83a14 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:48:48.266330 12030 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2ad6971df1e4cc688fb208e70d83a14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ad6971df1e4cc688fb208e70d83a14" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 44841 } }
I20260621 07:48:48.267936 12032 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2ad6971df1e4cc688fb208e70d83a14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2ad6971df1e4cc688fb208e70d83a14. Latest consensus state: current_term: 1 leader_uuid: "d2ad6971df1e4cc688fb208e70d83a14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ad6971df1e4cc688fb208e70d83a14" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 44841 } } }
I20260621 07:48:48.268067 12032 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2ad6971df1e4cc688fb208e70d83a14 [sys.catalog]: This master's current role is: LEADER
I20260621 07:48:48.268298 12025 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260621 07:48:48.268440 12031 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2ad6971df1e4cc688fb208e70d83a14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2ad6971df1e4cc688fb208e70d83a14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ad6971df1e4cc688fb208e70d83a14" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 44841 } } }
I20260621 07:48:48.268657 12031 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2ad6971df1e4cc688fb208e70d83a14 [sys.catalog]: This master's current role is: LEADER
I20260621 07:48:48.268534 12044 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:48:48.274597 12044 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:48:48.286154 12044 catalog_manager.cc:1347] Generated new cluster ID: 66495c43d8a94d188509592abf59a794
I20260621 07:48:48.286219 12044 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:48:48.301838 12044 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:48:48.303398 12044 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:48:48.316701 12044 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2ad6971df1e4cc688fb208e70d83a14: Generated new TSK 0
I20260621 07:48:48.317724 12044 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.192.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab.
Entry for principal kudu/127.0.192.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab.
I20260621 07:48:48.977721 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.192.212:54807/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.193:0
--local_ip_for_outbound_sockets=127.0.192.193
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:44841
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:40489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:48:49.106015 12070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:48:49.106246 12070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:48:49.106307 12070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:48:49.110059 12070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:48:49.110131 12070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:48:49.110225 12070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.193
I20260621 07:48:49.115438 12070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:40489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.192.212:54807/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:44841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.193
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:48:49.116356 12070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:48:49.117285 12070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:48:49.126842 12077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:48:49.127295 12078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:48:49.127784 12080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:48:49.128468 12070 server_base.cc:1034] running on GCE node
I20260621 07:48:49.128918 12070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:48:49.129644 12070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:48:49.130839 12070 hybrid_clock.cc:648] HybridClock initialized: now 1782028129130821 us; error 30 us; skew 500 ppm
Jun 21 07:48:49 dist-test-slave-lzx3 krb5kdc[11512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:48:49.134274 12070 init.cc:377] Logged in from keytab as kudu/127.0.192.193@KRBTEST.COM (short username kudu)
I20260621 07:48:49.135201 12070 webserver.cc:460] Webserver started at http://127.0.192.193:34831/ using document root <none> and password file <none>
I20260621 07:48:49.135828 12070 fs_manager.cc:359] Metadata directory not provided
I20260621 07:48:49.135905 12070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:48:49.136096 12070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 07:48:49 dist-test-slave-lzx3 krb5kdc[11512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for HTTP/127.0.192.212@KRBTEST.COM
I20260621 07:48:49.150331 12070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-0/data/instance:
uuid: "eac82b8522434baa85a487191284b23c"
format_stamp: "Formatted at 2026-06-21 07:48:49 on dist-test-slave-lzx3"
server_key: "c1f4090c9b2a2f3aaf830372ba800653"
server_key_iv: "0b1b219a8ecd153b69f45e2729e94ea2"
server_key_version: "kuduclusterkey@0"
I20260621 07:48:49.150893 12070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-0/wal/instance:
uuid: "eac82b8522434baa85a487191284b23c"
format_stamp: "Formatted at 2026-06-21 07:48:49 on dist-test-slave-lzx3"
server_key: "c1f4090c9b2a2f3aaf830372ba800653"
server_key_iv: "0b1b219a8ecd153b69f45e2729e94ea2"
server_key_version: "kuduclusterkey@0"
I20260621 07:48:49.154583 12070 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 07:48:49.184883 12087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:48:49.190932 12070 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.002s sys 0.001s
I20260621 07:48:49.191112 12070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-0/wal
uuid: "eac82b8522434baa85a487191284b23c"
format_stamp: "Formatted at 2026-06-21 07:48:49 on dist-test-slave-lzx3"
server_key: "c1f4090c9b2a2f3aaf830372ba800653"
server_key_iv: "0b1b219a8ecd153b69f45e2729e94ea2"
server_key_version: "kuduclusterkey@0"
I20260621 07:48:49.191355 12070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:48:49.216197 12070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:48:49.219717 12070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:48:49.219938 12070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:48:49.220654 12070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:48:49.221705 12070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:48:49.221809 12070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:48:49.221937 12070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:48:49.222020 12070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:48:49.232963 12070 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.193:42937
I20260621 07:48:49.233074 12202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.193:42937 every 8 connection(s)
I20260621 07:48:49.233907 12070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
I20260621 07:48:49.234810 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 12070
I20260621 07:48:49.234951 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-0/wal/instance
I20260621 07:48:49.235227 771 external_mini_cluster.cc:1428] Setting key ebde2326b100051085a9295890aa2c79
WARNING: no policy specified for kudu/127.0.192.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.194@KRBTEST.COM" created.
Jun 21 07:48:49 dist-test-slave-lzx3 krb5kdc[11512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:48:49.262660 12207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:49.240475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:49753 (local address 127.0.192.254:44841)
0621 07:48:49.246675 (+ 6200us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:49.246683 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:49.246696 (+ 13us) server_negotiation.cc:407] Connection header received
0621 07:48:49.246735 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:49.246738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:49.246780 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:49.246881 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:48:49.252694 (+ 5813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:49.253529 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:49.255088 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:49.255416 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:49.255488 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:49.258231 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:48:49.258237 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:48:49.258243 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:48:49.258273 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:48:49.260101 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:48:49.260746 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:48:49.260752 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:48:49.260755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:48:49.260848 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:48:49.261283 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:48:49.261289 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:48:49.261293 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:48:49.261520 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:48:49.261622 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0621 07:48:49.262271 (+ 649us) server_negotiation.cc:299] Negotiation successful
0621 07:48:49.262424 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6125,"thread_start_us":103,"threads_started":1}
I20260621 07:48:49.264223 12206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:49.239494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:44841 (local address 127.0.192.193:49753)
0621 07:48:49.239961 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0621 07:48:49.240010 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0621 07:48:49.241141 (+ 1131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:48:49.249895 (+ 8754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:48:49.249904 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:48:49.251845 (+ 1941us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:48:49.252473 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:49.252490 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:49.253679 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:49.253683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:49.254965 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:49.254974 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:49.255672 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:49.255676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:49.255856 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:49.256446 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:48:49.256460 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:48:49.258049 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:48:49.260230 (+ 2181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:48:49.260237 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:48:49.260254 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:48:49.260626 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:48:49.260962 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:48:49.260966 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:48:49.260969 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:48:49.261144 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:48:49.261700 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:48:49.261706 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:48:49.261994 (+ 288us) client_negotiation.cc:769] Sending connection context
0621 07:48:49.262917 (+ 923us) client_negotiation.cc:240] Negotiation successful
0621 07:48:49.263231 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":153,"threads_started":1}
I20260621 07:48:49.265455 12203 heartbeater.cc:347] Connected to a master server at 127.0.192.254:44841
I20260621 07:48:49.265664 12203 heartbeater.cc:464] Registering TS with master...
I20260621 07:48:49.266170 12203 heartbeater.cc:511] Master 127.0.192.254:44841 requested a full tablet report, sending...
I20260621 07:48:49.267776 11989 ts_manager.cc:194] Registered new tserver with Master: eac82b8522434baa85a487191284b23c (127.0.192.193:42937)
I20260621 07:48:49.268983 11989 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.193@KRBTEST.COM'} at 127.0.192.193:49753
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab.
Entry for principal kudu/127.0.192.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab.
I20260621 07:48:49.273592 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.192.212:54807/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.194:0
--local_ip_for_outbound_sockets=127.0.192.194
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:44841
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:40489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 07:48:49 dist-test-slave-lzx3 krb5kdc[11512](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782028129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260621 07:48:49.387076 12209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:48:49.387353 12209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:48:49.387463 12209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:48:49.391353 12209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:48:49.391430 12209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:48:49.391517 12209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.194
I20260621 07:48:49.396524 12209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:40489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.192.212:54807/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:44841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.194
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:48:49.397461 12209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:48:49.398403 12209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:48:49.406782 12215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:48:49.406836 12216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:48:49.407167 12218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:48:49.407274 12209 server_base.cc:1034] running on GCE node
I20260621 07:48:49.407666 12209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:48:49.408311 12209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:48:49.415704 12209 hybrid_clock.cc:648] HybridClock initialized: now 1782028129413941 us; error 1774 us; skew 500 ppm
Jun 21 07:48:49 dist-test-slave-lzx3 krb5kdc[11512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:48:49.418839 12209 init.cc:377] Logged in from keytab as kudu/127.0.192.194@KRBTEST.COM (short username kudu)
I20260621 07:48:49.419739 12209 webserver.cc:460] Webserver started at http://127.0.192.194:36573/ using document root <none> and password file <none>
I20260621 07:48:49.420249 12209 fs_manager.cc:359] Metadata directory not provided
I20260621 07:48:49.420324 12209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:48:49.420511 12209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 07:48:49 dist-test-slave-lzx3 krb5kdc[11512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for HTTP/127.0.192.212@KRBTEST.COM
I20260621 07:48:49.440631 12209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-1/data/instance:
uuid: "c7d4911794b645378672dfe92ffb7ae1"
format_stamp: "Formatted at 2026-06-21 07:48:49 on dist-test-slave-lzx3"
server_key: "1a7c6ad9f38b04bd90d0870f2f2854d4"
server_key_iv: "9a0032cc86d766c9014f64d10bc81acd"
server_key_version: "kuduclusterkey@0"
I20260621 07:48:49.441411 12209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-1/wal/instance:
uuid: "c7d4911794b645378672dfe92ffb7ae1"
format_stamp: "Formatted at 2026-06-21 07:48:49 on dist-test-slave-lzx3"
server_key: "1a7c6ad9f38b04bd90d0870f2f2854d4"
server_key_iv: "9a0032cc86d766c9014f64d10bc81acd"
server_key_version: "kuduclusterkey@0"
I20260621 07:48:49.446802 12209 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.002s
I20260621 07:48:49.466867 12225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:48:49.468516 12209 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.001s
I20260621 07:48:49.468665 12209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-1/wal
uuid: "c7d4911794b645378672dfe92ffb7ae1"
format_stamp: "Formatted at 2026-06-21 07:48:49 on dist-test-slave-lzx3"
server_key: "1a7c6ad9f38b04bd90d0870f2f2854d4"
server_key_iv: "9a0032cc86d766c9014f64d10bc81acd"
server_key_version: "kuduclusterkey@0"
I20260621 07:48:49.468796 12209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:48:49.492571 12209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:48:49.495858 12209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:48:49.496042 12209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:48:49.496647 12209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:48:49.497599 12209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:48:49.497697 12209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:48:49.497778 12209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:48:49.497831 12209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:48:49.507345 12209 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.194:33179
I20260621 07:48:49.508337 12209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
I20260621 07:48:49.509809 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 12209
I20260621 07:48:49.509929 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-1/wal/instance
I20260621 07:48:49.510617 771 external_mini_cluster.cc:1428] Setting key 305640f3d9a12e97bafaad2505027efe
I20260621 07:48:49.513278 12338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.194:33179 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.192.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.195@KRBTEST.COM" created.
Jun 21 07:48:49 dist-test-slave-lzx3 krb5kdc[11512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:48:49.541800 12207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:49.516410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:34243 (local address 127.0.192.254:44841)
0621 07:48:49.516524 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:49.516529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:49.522367 (+ 5838us) server_negotiation.cc:407] Connection header received
0621 07:48:49.523698 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:49.523701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:49.523749 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:49.523821 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:48:49.527040 (+ 3219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:49.527812 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:49.529361 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:49.529696 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:49.529802 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:49.532729 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:48:49.532733 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:48:49.532735 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:48:49.532759 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:48:49.534242 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:48:49.534821 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:48:49.534825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:48:49.534826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:48:49.534923 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:48:49.535502 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:48:49.535505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:48:49.535507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:48:49.535737 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:48:49.535822 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0621 07:48:49.541471 (+ 5649us) server_negotiation.cc:299] Negotiation successful
0621 07:48:49.541616 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260621 07:48:49.543690 12342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:49.515723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:44841 (local address 127.0.192.194:34243)
0621 07:48:49.522208 (+ 6485us) negotiation.cc:107] Waiting for socket to connect
0621 07:48:49.522266 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0621 07:48:49.523465 (+ 1199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:48:49.524213 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:48:49.524221 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:48:49.524513 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:48:49.526854 (+ 2341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:49.526875 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:49.527963 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:49.527966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:49.529237 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:49.529246 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:49.529962 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:49.529965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:49.530144 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:49.530759 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:48:49.530779 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:48:49.532518 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:48:49.534375 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:48:49.534380 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:48:49.534394 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:48:49.534709 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:48:49.535210 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:48:49.535213 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:48:49.535216 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:48:49.535386 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:48:49.535885 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:48:49.535890 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:48:49.536218 (+ 328us) client_negotiation.cc:769] Sending connection context
0621 07:48:49.542289 (+ 6071us) client_negotiation.cc:240] Negotiation successful
0621 07:48:49.542669 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6234,"thread_start_us":151,"threads_started":1}
I20260621 07:48:49.545095 12339 heartbeater.cc:347] Connected to a master server at 127.0.192.254:44841
I20260621 07:48:49.545311 12339 heartbeater.cc:464] Registering TS with master...
I20260621 07:48:49.545768 12339 heartbeater.cc:511] Master 127.0.192.254:44841 requested a full tablet report, sending...
I20260621 07:48:49.546655 11989 ts_manager.cc:194] Registered new tserver with Master: c7d4911794b645378672dfe92ffb7ae1 (127.0.192.194:33179)
I20260621 07:48:49.547201 11989 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.194@KRBTEST.COM'} at 127.0.192.194:34243
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab.
Entry for principal kudu/127.0.192.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab.
I20260621 07:48:49.549726 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.192.212:54807/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.195:0
--local_ip_for_outbound_sockets=127.0.192.195
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:44841
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:40489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:48:49.691205 12344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:48:49.691514 12344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:48:49.691571 12344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:48:49.696771 12344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:48:49.696857 12344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:48:49.696945 12344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.195
I20260621 07:48:49.703648 12344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:40489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.192.212:54807/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:44841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.195
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:48:49.704823 12344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:48:49.705912 12344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:48:49.716640 12352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:48:49.717491 12344 server_base.cc:1034] running on GCE node
W20260621 07:48:49.718022 12350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:48:49.718251 12349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:48:49.718711 12344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:48:49.719596 12344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:48:49.720752 12344 hybrid_clock.cc:648] HybridClock initialized: now 1782028129720687 us; error 79 us; skew 500 ppm
Jun 21 07:48:49 dist-test-slave-lzx3 krb5kdc[11512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:48:49.733474 12344 init.cc:377] Logged in from keytab as kudu/127.0.192.195@KRBTEST.COM (short username kudu)
I20260621 07:48:49.734529 12344 webserver.cc:460] Webserver started at http://127.0.192.195:41025/ using document root <none> and password file <none>
I20260621 07:48:49.735198 12344 fs_manager.cc:359] Metadata directory not provided
I20260621 07:48:49.735286 12344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:48:49.735488 12344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 07:48:49 dist-test-slave-lzx3 krb5kdc[11512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for HTTP/127.0.192.212@KRBTEST.COM
I20260621 07:48:49.758388 12344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-2/data/instance:
uuid: "aabe6ac224074bf2ad5e686897992b1e"
format_stamp: "Formatted at 2026-06-21 07:48:49 on dist-test-slave-lzx3"
server_key: "0bbd04882f3c09495ed5be8afdff99c8"
server_key_iv: "85dce2679db98c4cc63924f22ac2e701"
server_key_version: "kuduclusterkey@0"
I20260621 07:48:49.759109 12344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-2/wal/instance:
uuid: "aabe6ac224074bf2ad5e686897992b1e"
format_stamp: "Formatted at 2026-06-21 07:48:49 on dist-test-slave-lzx3"
server_key: "0bbd04882f3c09495ed5be8afdff99c8"
server_key_iv: "85dce2679db98c4cc63924f22ac2e701"
server_key_version: "kuduclusterkey@0"
Jun 21 07:48:49 dist-test-slave-lzx3 krb5kdc[11512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for HTTP/127.0.192.212@KRBTEST.COM
I20260621 07:48:49.764585 12344 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 07:48:49.800866 12359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:48:49.808287 12344 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.003s sys 0.000s
I20260621 07:48:49.808436 12344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-2/wal
uuid: "aabe6ac224074bf2ad5e686897992b1e"
format_stamp: "Formatted at 2026-06-21 07:48:49 on dist-test-slave-lzx3"
server_key: "0bbd04882f3c09495ed5be8afdff99c8"
server_key_iv: "85dce2679db98c4cc63924f22ac2e701"
server_key_version: "kuduclusterkey@0"
I20260621 07:48:49.808619 12344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:48:49.831223 12344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:48:49.843292 12344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:48:49.843488 12344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:48:49.844163 12344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:48:49.845492 12344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:48:49.845535 12344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:48:49.845582 12344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:48:49.845599 12344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:48:49.855268 12344 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.195:43715
I20260621 07:48:49.864095 12344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
I20260621 07:48:49.865859 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 12344
I20260621 07:48:49.865969 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-2/wal/instance
I20260621 07:48:49.866286 771 external_mini_cluster.cc:1428] Setting key 21972ea20516236374ff94a0d7d5b3e2
I20260621 07:48:49.865741 12472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.195:43715 every 8 connection(s)
Jun 21 07:48:49 dist-test-slave-lzx3 krb5kdc[11512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:48:49.884341 12207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:49.870055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:52015 (local address 127.0.192.254:44841)
0621 07:48:49.870182 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:49.870187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:49.872094 (+ 1907us) server_negotiation.cc:407] Connection header received
0621 07:48:49.873231 (+ 1137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:49.873235 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:49.873279 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:49.873320 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:48:49.874679 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:49.875410 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:49.876858 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:49.877177 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:49.877241 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:49.879987 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:48:49.879991 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:48:49.879994 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:48:49.880018 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:48:49.881673 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:48:49.882350 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:48:49.882354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:48:49.882356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:48:49.882472 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:48:49.882925 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:48:49.882928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:48:49.882930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:48:49.883140 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:48:49.883234 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0621 07:48:49.883951 (+ 717us) server_negotiation.cc:299] Negotiation successful
0621 07:48:49.884074 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260621 07:48:49.886094 12475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:49.870541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:44841 (local address 127.0.192.195:52015)
0621 07:48:49.871948 (+ 1407us) negotiation.cc:107] Waiting for socket to connect
0621 07:48:49.872006 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0621 07:48:49.873012 (+ 1006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:48:49.873659 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:48:49.873667 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:48:49.873971 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:48:49.874519 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:49.874535 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:49.875540 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:49.875543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:49.876743 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:49.876752 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:49.877410 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:49.877413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:49.877586 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:49.878259 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:48:49.878289 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:48:49.879796 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:48:49.881820 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:48:49.881826 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:48:49.881840 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:48:49.882212 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:48:49.882612 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:48:49.882615 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:48:49.882618 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:48:49.882800 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:48:49.883324 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:48:49.883331 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:48:49.883657 (+ 326us) client_negotiation.cc:769] Sending connection context
0621 07:48:49.884795 (+ 1138us) client_negotiation.cc:240] Negotiation successful
0621 07:48:49.885112 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1191,"thread_start_us":128,"threads_started":1}
I20260621 07:48:49.887521 12473 heartbeater.cc:347] Connected to a master server at 127.0.192.254:44841
I20260621 07:48:49.887753 12473 heartbeater.cc:464] Registering TS with master...
I20260621 07:48:49.888293 12473 heartbeater.cc:511] Master 127.0.192.254:44841 requested a full tablet report, sending...
I20260621 07:48:49.889343 11989 ts_manager.cc:194] Registered new tserver with Master: aabe6ac224074bf2ad5e686897992b1e (127.0.192.195:43715)
I20260621 07:48:49.890007 11989 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.195@KRBTEST.COM'} at 127.0.192.195:52015
I20260621 07:48:49.893235 771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:48:49.905014 12207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:49.895548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60450 (local address 127.0.192.254:44841)
0621 07:48:49.895650 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:49.895654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:49.896259 (+ 605us) server_negotiation.cc:407] Connection header received
0621 07:48:49.896417 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:49.896420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:49.896459 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:49.896504 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:48:49.897450 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:49.898248 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:49.899425 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:49.899732 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:49.899796 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:49.901510 (+ 1714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:48:49.901513 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:48:49.901516 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:48:49.901539 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:48:49.903202 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:48:49.903679 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:48:49.903683 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:48:49.903685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:48:49.903801 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:48:49.904080 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:48:49.904083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:48:49.904085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:48:49.904276 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:48:49.904370 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0621 07:48:49.904742 (+ 372us) server_negotiation.cc:299] Negotiation successful
0621 07:48:49.904865 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260621 07:48:49.908643 11989 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:48:49.912822 11989 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:48:49.938061 12486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:49.925513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:42937 (local address 127.0.0.1:39848)
0621 07:48:49.928660 (+ 3147us) negotiation.cc:107] Waiting for socket to connect
0621 07:48:49.928683 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 07:48:49.928798 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:48:49.929496 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:48:49.929502 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:48:49.929520 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:48:49.929654 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:49.929660 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:49.930826 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:49.930829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:49.932087 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:49.932097 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:49.937595 (+ 5498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:49.937598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:49.937702 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:49.937734 (+ 32us) client_negotiation.cc:769] Sending connection context
0621 07:48:49.937826 (+ 92us) client_negotiation.cc:240] Negotiation successful
0621 07:48:49.937922 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3106,"thread_start_us":76,"threads_started":1}
I20260621 07:48:49.938513 12490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:49.924534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:33179 (local address 127.0.0.1:46886)
0621 07:48:49.928639 (+ 4105us) negotiation.cc:107] Waiting for socket to connect
0621 07:48:49.928684 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 07:48:49.930376 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:48:49.930893 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:48:49.930895 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:48:49.930909 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:48:49.930968 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:49.930973 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:49.935808 (+ 4835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:49.935811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:49.936827 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:49.936835 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:49.937848 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:49.937851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:49.937914 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:49.937930 (+ 16us) client_negotiation.cc:769] Sending connection context
0621 07:48:49.937985 (+ 55us) client_negotiation.cc:240] Negotiation successful
0621 07:48:49.938028 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3930,"thread_start_us":126,"threads_started":1}
I20260621 07:48:49.938946 12489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:49.926418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39848 (local address 127.0.192.193:42937)
0621 07:48:49.928395 (+ 1977us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:49.928400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:49.928925 (+ 525us) server_negotiation.cc:407] Connection header received
0621 07:48:49.928997 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:49.929002 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:49.929198 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:49.929343 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:48:49.929763 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:49.930727 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:49.936635 (+ 5908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:49.937322 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:49.937412 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:49.937538 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0621 07:48:49.938730 (+ 1192us) server_negotiation.cc:299] Negotiation successful
0621 07:48:49.938827 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1893,"thread_start_us":72,"threads_started":1}
I20260621 07:48:49.941979 12125 tablet_service.cc:1467] Processing CreateTablet for tablet 2a689150b07446de8b12a54f69b7041f (DEFAULT_TABLE table=test-table [id=45c9b13ca98a44ef88b107586754c1e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:48:49.943142 12125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a689150b07446de8b12a54f69b7041f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:48:49.953403 12487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:49.927485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.195:43715 (local address 127.0.0.1:33384)
0621 07:48:49.929530 (+ 2045us) negotiation.cc:107] Waiting for socket to connect
0621 07:48:49.929542 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:48:49.929610 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:48:49.930170 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:48:49.930172 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:48:49.930186 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:48:49.930253 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:49.930257 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:49.933086 (+ 2829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:49.933089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:49.934048 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:49.934058 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:49.951678 (+ 17620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:49.951682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:49.951781 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:49.951803 (+ 22us) client_negotiation.cc:769] Sending connection context
0621 07:48:49.951877 (+ 74us) client_negotiation.cc:240] Negotiation successful
0621 07:48:49.951946 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2000,"thread_start_us":62,"threads_started":1}
I20260621 07:48:49.955597 12488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:49.926126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46886 (local address 127.0.192.194:33179)
0621 07:48:49.928235 (+ 2109us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:49.928246 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:49.930442 (+ 2196us) server_negotiation.cc:407] Connection header received
0621 07:48:49.930506 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:49.930510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:49.930648 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:49.930776 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:48:49.931068 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:49.935676 (+ 4608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:49.936952 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:49.937587 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:49.937669 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:49.937799 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0621 07:48:49.955353 (+ 17554us) server_negotiation.cc:299] Negotiation successful
0621 07:48:49.955470 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2014,"thread_start_us":89,"threads_started":1}
I20260621 07:48:49.956036 12491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:49.927786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33384 (local address 127.0.192.195:43715)
0621 07:48:49.929418 (+ 1632us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:49.929428 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:49.929687 (+ 259us) server_negotiation.cc:407] Connection header received
0621 07:48:49.929770 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:49.929774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:49.929933 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:49.930058 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:48:49.931241 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:49.932970 (+ 1729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:49.938123 (+ 5153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:49.951300 (+ 13177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:49.951417 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:49.951573 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0621 07:48:49.955817 (+ 4244us) server_negotiation.cc:299] Negotiation successful
0621 07:48:49.955912 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1552,"thread_start_us":89,"threads_started":1}
I20260621 07:48:49.959151 12273 tablet_service.cc:1467] Processing CreateTablet for tablet 2a689150b07446de8b12a54f69b7041f (DEFAULT_TABLE table=test-table [id=45c9b13ca98a44ef88b107586754c1e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:48:49.960096 12273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a689150b07446de8b12a54f69b7041f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:48:49.969487 12407 tablet_service.cc:1467] Processing CreateTablet for tablet 2a689150b07446de8b12a54f69b7041f (DEFAULT_TABLE table=test-table [id=45c9b13ca98a44ef88b107586754c1e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:48:49.970435 12407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a689150b07446de8b12a54f69b7041f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:48:49.979871 12493 tablet_bootstrap.cc:492] T 2a689150b07446de8b12a54f69b7041f P aabe6ac224074bf2ad5e686897992b1e: Bootstrap starting.
I20260621 07:48:49.980232 12494 tablet_bootstrap.cc:492] T 2a689150b07446de8b12a54f69b7041f P c7d4911794b645378672dfe92ffb7ae1: Bootstrap starting.
I20260621 07:48:49.983807 12492 tablet_bootstrap.cc:492] T 2a689150b07446de8b12a54f69b7041f P eac82b8522434baa85a487191284b23c: Bootstrap starting.
I20260621 07:48:49.985411 12493 tablet_bootstrap.cc:654] T 2a689150b07446de8b12a54f69b7041f P aabe6ac224074bf2ad5e686897992b1e: Neither blocks nor log segments found. Creating new log.
I20260621 07:48:49.987409 12492 tablet_bootstrap.cc:654] T 2a689150b07446de8b12a54f69b7041f P eac82b8522434baa85a487191284b23c: Neither blocks nor log segments found. Creating new log.
I20260621 07:48:49.988785 12494 tablet_bootstrap.cc:654] T 2a689150b07446de8b12a54f69b7041f P c7d4911794b645378672dfe92ffb7ae1: Neither blocks nor log segments found. Creating new log.
I20260621 07:48:49.989609 12493 log.cc:824] T 2a689150b07446de8b12a54f69b7041f P aabe6ac224074bf2ad5e686897992b1e: Log is configured to *not* fsync() on all Append() calls
I20260621 07:48:49.990561 12494 log.cc:824] T 2a689150b07446de8b12a54f69b7041f P c7d4911794b645378672dfe92ffb7ae1: Log is configured to *not* fsync() on all Append() calls
I20260621 07:48:49.991266 12492 log.cc:824] T 2a689150b07446de8b12a54f69b7041f P eac82b8522434baa85a487191284b23c: Log is configured to *not* fsync() on all Append() calls
I20260621 07:48:49.994338 12494 tablet_bootstrap.cc:492] T 2a689150b07446de8b12a54f69b7041f P c7d4911794b645378672dfe92ffb7ae1: No bootstrap required, opened a new log
I20260621 07:48:49.994699 12494 ts_tablet_manager.cc:1397] T 2a689150b07446de8b12a54f69b7041f P c7d4911794b645378672dfe92ffb7ae1: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.004s
I20260621 07:48:49.999276 12493 tablet_bootstrap.cc:492] T 2a689150b07446de8b12a54f69b7041f P aabe6ac224074bf2ad5e686897992b1e: No bootstrap required, opened a new log
I20260621 07:48:49.999501 12493 ts_tablet_manager.cc:1397] T 2a689150b07446de8b12a54f69b7041f P aabe6ac224074bf2ad5e686897992b1e: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.000s
I20260621 07:48:50.003069 12493 raft_consensus.cc:348] T 2a689150b07446de8b12a54f69b7041f P aabe6ac224074bf2ad5e686897992b1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eac82b8522434baa85a487191284b23c" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 42937 } } peers { permanent_uuid: "c7d4911794b645378672dfe92ffb7ae1" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 33179 } } peers { permanent_uuid: "aabe6ac224074bf2ad5e686897992b1e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43715 } }
I20260621 07:48:50.005085 12494 raft_consensus.cc:348] T 2a689150b07446de8b12a54f69b7041f P c7d4911794b645378672dfe92ffb7ae1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eac82b8522434baa85a487191284b23c" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 42937 } } peers { permanent_uuid: "c7d4911794b645378672dfe92ffb7ae1" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 33179 } } peers { permanent_uuid: "aabe6ac224074bf2ad5e686897992b1e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43715 } }
I20260621 07:48:50.005348 12493 raft_consensus.cc:374] T 2a689150b07446de8b12a54f69b7041f P aabe6ac224074bf2ad5e686897992b1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:48:50.005429 12493 raft_consensus.cc:729] T 2a689150b07446de8b12a54f69b7041f P aabe6ac224074bf2ad5e686897992b1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aabe6ac224074bf2ad5e686897992b1e, State: Initialized, Role: FOLLOWER
I20260621 07:48:50.005944 12493 consensus_queue.cc:260] T 2a689150b07446de8b12a54f69b7041f P aabe6ac224074bf2ad5e686897992b1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eac82b8522434baa85a487191284b23c" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 42937 } } peers { permanent_uuid: "c7d4911794b645378672dfe92ffb7ae1" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 33179 } } peers { permanent_uuid: "aabe6ac224074bf2ad5e686897992b1e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43715 } }
I20260621 07:48:50.006177 12494 raft_consensus.cc:374] T 2a689150b07446de8b12a54f69b7041f P c7d4911794b645378672dfe92ffb7ae1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:48:50.007086 12493 ts_tablet_manager.cc:1428] T 2a689150b07446de8b12a54f69b7041f P aabe6ac224074bf2ad5e686897992b1e: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260621 07:48:50.007200 12494 raft_consensus.cc:729] T 2a689150b07446de8b12a54f69b7041f P c7d4911794b645378672dfe92ffb7ae1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7d4911794b645378672dfe92ffb7ae1, State: Initialized, Role: FOLLOWER
I20260621 07:48:50.007634 12494 consensus_queue.cc:260] T 2a689150b07446de8b12a54f69b7041f P c7d4911794b645378672dfe92ffb7ae1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eac82b8522434baa85a487191284b23c" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 42937 } } peers { permanent_uuid: "c7d4911794b645378672dfe92ffb7ae1" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 33179 } } peers { permanent_uuid: "aabe6ac224074bf2ad5e686897992b1e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43715 } }
I20260621 07:48:50.008464 12494 ts_tablet_manager.cc:1428] T 2a689150b07446de8b12a54f69b7041f P c7d4911794b645378672dfe92ffb7ae1: Time spent starting tablet: real 0.014s user 0.002s sys 0.003s
I20260621 07:48:50.009843 12492 tablet_bootstrap.cc:492] T 2a689150b07446de8b12a54f69b7041f P eac82b8522434baa85a487191284b23c: No bootstrap required, opened a new log
I20260621 07:48:50.010098 12492 ts_tablet_manager.cc:1397] T 2a689150b07446de8b12a54f69b7041f P eac82b8522434baa85a487191284b23c: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20260621 07:48:50.015682 12492 raft_consensus.cc:348] T 2a689150b07446de8b12a54f69b7041f P eac82b8522434baa85a487191284b23c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eac82b8522434baa85a487191284b23c" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 42937 } } peers { permanent_uuid: "c7d4911794b645378672dfe92ffb7ae1" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 33179 } } peers { permanent_uuid: "aabe6ac224074bf2ad5e686897992b1e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43715 } }
I20260621 07:48:50.015961 12492 raft_consensus.cc:374] T 2a689150b07446de8b12a54f69b7041f P eac82b8522434baa85a487191284b23c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:48:50.016041 12492 raft_consensus.cc:729] T 2a689150b07446de8b12a54f69b7041f P eac82b8522434baa85a487191284b23c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eac82b8522434baa85a487191284b23c, State: Initialized, Role: FOLLOWER
I20260621 07:48:50.016448 12492 consensus_queue.cc:260] T 2a689150b07446de8b12a54f69b7041f P eac82b8522434baa85a487191284b23c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eac82b8522434baa85a487191284b23c" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 42937 } } peers { permanent_uuid: "c7d4911794b645378672dfe92ffb7ae1" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 33179 } } peers { permanent_uuid: "aabe6ac224074bf2ad5e686897992b1e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43715 } }
I20260621 07:48:50.017623 12492 ts_tablet_manager.cc:1428] T 2a689150b07446de8b12a54f69b7041f P eac82b8522434baa85a487191284b23c: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260621 07:48:50.017863 12473 heartbeater.cc:503] Master 127.0.192.254:44841 was elected leader, sending a full tablet report...
W20260621 07:48:50.015443 12340 tablet.cc:2369] T 2a689150b07446de8b12a54f69b7041f P c7d4911794b645378672dfe92ffb7ae1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:48:50.021852 12339 heartbeater.cc:503] Master 127.0.192.254:44841 was elected leader, sending a full tablet report...
I20260621 07:48:50.022017 12203 heartbeater.cc:503] Master 127.0.192.254:44841 was elected leader, sending a full tablet report...
Jun 21 07:48:50 dist-test-slave-lzx3 krb5kdc[11512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for HTTP/127.0.192.212@KRBTEST.COM
W20260621 07:48:50.121941 12474 tablet.cc:2369] T 2a689150b07446de8b12a54f69b7041f P aabe6ac224074bf2ad5e686897992b1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 07:48:50.248011 12204 tablet.cc:2369] T 2a689150b07446de8b12a54f69b7041f P eac82b8522434baa85a487191284b23c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:48:50.293272 12500 raft_consensus.cc:482] T 2a689150b07446de8b12a54f69b7041f P eac82b8522434baa85a487191284b23c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:48:50.293529 12500 raft_consensus.cc:504] T 2a689150b07446de8b12a54f69b7041f P eac82b8522434baa85a487191284b23c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eac82b8522434baa85a487191284b23c" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 42937 } } peers { permanent_uuid: "c7d4911794b645378672dfe92ffb7ae1" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 33179 } } peers { permanent_uuid: "aabe6ac224074bf2ad5e686897992b1e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43715 } }
I20260621 07:48:50.295253 12500 leader_election.cc:288] T 2a689150b07446de8b12a54f69b7041f P eac82b8522434baa85a487191284b23c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7d4911794b645378672dfe92ffb7ae1 (127.0.192.194:33179), aabe6ac224074bf2ad5e686897992b1e (127.0.192.195:43715)
I20260621 07:48:50.304234 12491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:50.296170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:52399 (local address 127.0.192.195:43715)
0621 07:48:50.296289 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:50.296295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:50.299861 (+ 3566us) server_negotiation.cc:407] Connection header received
0621 07:48:50.299970 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:50.299974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:50.300031 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:50.300117 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:48:50.300541 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:50.301350 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:50.303035 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:50.303643 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:50.303767 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:50.303840 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0621 07:48:50.304055 (+ 215us) server_negotiation.cc:299] Negotiation successful
0621 07:48:50.304118 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260621 07:48:50.304265 12505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:50.295681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.195:43715 (local address 127.0.192.193:52399)
0621 07:48:50.299742 (+ 4061us) negotiation.cc:107] Waiting for socket to connect
0621 07:48:50.299759 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:48:50.299877 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:48:50.300250 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:48:50.300256 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:48:50.300280 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:48:50.300400 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:50.300407 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:50.301500 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:50.301503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:50.302881 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:50.302894 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:50.303822 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:50.303826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:50.303914 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:50.303939 (+ 25us) client_negotiation.cc:769] Sending connection context
0621 07:48:50.304031 (+ 92us) client_negotiation.cc:240] Negotiation successful
0621 07:48:50.304100 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3988,"mutex_wait_us":55,"thread_start_us":198,"threads_started":1}
I20260621 07:48:50.304502 12488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:50.296908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:34771 (local address 127.0.192.194:33179)
0621 07:48:50.297023 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:50.297028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:50.300141 (+ 3113us) server_negotiation.cc:407] Connection header received
0621 07:48:50.300325 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:50.300328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:50.300374 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:50.300419 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:48:50.300820 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:50.301605 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:50.303185 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:50.303796 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:50.303891 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:50.303937 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0621 07:48:50.304296 (+ 359us) server_negotiation.cc:299] Negotiation successful
0621 07:48:50.304384 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260621 07:48:50.304888 12506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:50.296497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:33179 (local address 127.0.192.193:34771)
0621 07:48:50.300066 (+ 3569us) negotiation.cc:107] Waiting for socket to connect
0621 07:48:50.300077 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 07:48:50.300202 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:48:50.300542 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:48:50.300545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:48:50.300582 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:48:50.300698 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:50.300703 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:50.301745 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:50.301748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:50.303044 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:48:50.303055 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:48:50.304038 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:48:50.304042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:48:50.304143 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:50.304165 (+ 22us) client_negotiation.cc:769] Sending connection context
0621 07:48:50.304668 (+ 503us) client_negotiation.cc:240] Negotiation successful
0621 07:48:50.304737 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3512,"mutex_wait_us":23,"thread_start_us":115,"threads_started":1}
I20260621 07:48:50.304999 12427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a689150b07446de8b12a54f69b7041f" candidate_uuid: "eac82b8522434baa85a487191284b23c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aabe6ac224074bf2ad5e686897992b1e" is_pre_election: true
I20260621 07:48:50.305333 12427 raft_consensus.cc:2454] T 2a689150b07446de8b12a54f69b7041f P aabe6ac224074bf2ad5e686897992b1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eac82b8522434baa85a487191284b23c in term 0.
I20260621 07:48:50.305521 12293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a689150b07446de8b12a54f69b7041f" candidate_uuid: "eac82b8522434baa85a487191284b23c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7d4911794b645378672dfe92ffb7ae1" is_pre_election: true
I20260621 07:48:50.305800 12293 raft_consensus.cc:2454] T 2a689150b07446de8b12a54f69b7041f P c7d4911794b645378672dfe92ffb7ae1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eac82b8522434baa85a487191284b23c in term 0.
I20260621 07:48:50.305925 12090 leader_election.cc:302] T 2a689150b07446de8b12a54f69b7041f P eac82b8522434baa85a487191284b23c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aabe6ac224074bf2ad5e686897992b1e, eac82b8522434baa85a487191284b23c; no voters:
I20260621 07:48:50.306345 12500 raft_consensus.cc:2781] T 2a689150b07446de8b12a54f69b7041f P eac82b8522434baa85a487191284b23c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:48:50.306437 12500 raft_consensus.cc:482] T 2a689150b07446de8b12a54f69b7041f P eac82b8522434baa85a487191284b23c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:48:50.306492 12500 raft_consensus.cc:3037] T 2a689150b07446de8b12a54f69b7041f P eac82b8522434baa85a487191284b23c [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:48:50.307502 12500 raft_consensus.cc:504] T 2a689150b07446de8b12a54f69b7041f P eac82b8522434baa85a487191284b23c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eac82b8522434baa85a487191284b23c" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 42937 } } peers { permanent_uuid: "c7d4911794b645378672dfe92ffb7ae1" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 33179 } } peers { permanent_uuid: "aabe6ac224074bf2ad5e686897992b1e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43715 } }
I20260621 07:48:50.307888 12500 leader_election.cc:288] T 2a689150b07446de8b12a54f69b7041f P eac82b8522434baa85a487191284b23c [CANDIDATE]: Term 1 election: Requested vote from peers c7d4911794b645378672dfe92ffb7ae1 (127.0.192.194:33179), aabe6ac224074bf2ad5e686897992b1e (127.0.192.195:43715)
I20260621 07:48:50.308140 12293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a689150b07446de8b12a54f69b7041f" candidate_uuid: "eac82b8522434baa85a487191284b23c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7d4911794b645378672dfe92ffb7ae1"
I20260621 07:48:50.308216 12427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a689150b07446de8b12a54f69b7041f" candidate_uuid: "eac82b8522434baa85a487191284b23c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aabe6ac224074bf2ad5e686897992b1e"
I20260621 07:48:50.308297 12293 raft_consensus.cc:3037] T 2a689150b07446de8b12a54f69b7041f P c7d4911794b645378672dfe92ffb7ae1 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:48:50.308365 12427 raft_consensus.cc:3037] T 2a689150b07446de8b12a54f69b7041f P aabe6ac224074bf2ad5e686897992b1e [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:48:50.309463 12427 raft_consensus.cc:2454] T 2a689150b07446de8b12a54f69b7041f P aabe6ac224074bf2ad5e686897992b1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eac82b8522434baa85a487191284b23c in term 1.
I20260621 07:48:50.309571 12293 raft_consensus.cc:2454] T 2a689150b07446de8b12a54f69b7041f P c7d4911794b645378672dfe92ffb7ae1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eac82b8522434baa85a487191284b23c in term 1.
I20260621 07:48:50.309754 12090 leader_election.cc:302] T 2a689150b07446de8b12a54f69b7041f P eac82b8522434baa85a487191284b23c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aabe6ac224074bf2ad5e686897992b1e, eac82b8522434baa85a487191284b23c; no voters:
I20260621 07:48:50.309970 12500 raft_consensus.cc:2781] T 2a689150b07446de8b12a54f69b7041f P eac82b8522434baa85a487191284b23c [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:48:50.310184 12500 raft_consensus.cc:686] T 2a689150b07446de8b12a54f69b7041f P eac82b8522434baa85a487191284b23c [term 1 LEADER]: Becoming Leader. State: Replica: eac82b8522434baa85a487191284b23c, State: Running, Role: LEADER
I20260621 07:48:50.310581 12500 consensus_queue.cc:237] T 2a689150b07446de8b12a54f69b7041f P eac82b8522434baa85a487191284b23c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eac82b8522434baa85a487191284b23c" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 42937 } } peers { permanent_uuid: "c7d4911794b645378672dfe92ffb7ae1" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 33179 } } peers { permanent_uuid: "aabe6ac224074bf2ad5e686897992b1e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43715 } }
I20260621 07:48:50.313871 11987 catalog_manager.cc:5495] T 2a689150b07446de8b12a54f69b7041f P eac82b8522434baa85a487191284b23c reported cstate change: term changed from 0 to 1, leader changed from <none> to eac82b8522434baa85a487191284b23c (127.0.192.193). New cstate: current_term: 1 leader_uuid: "eac82b8522434baa85a487191284b23c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eac82b8522434baa85a487191284b23c" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c7d4911794b645378672dfe92ffb7ae1" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 33179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aabe6ac224074bf2ad5e686897992b1e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43715 } health_report { overall_health: UNKNOWN } } }
I20260621 07:48:50.388878 12489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:50.385757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39852 (local address 127.0.192.193:42937)
0621 07:48:50.385885 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0621 07:48:50.385889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:48:50.385902 (+ 13us) server_negotiation.cc:407] Connection header received
0621 07:48:50.386009 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:48:50.386011 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:48:50.386064 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:48:50.386150 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:48:50.386357 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:50.386840 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:50.387607 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:48:50.387810 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:48:50.387882 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:48:50.388124 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:48:50.388495 (+ 371us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:48:50.388599 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0621 07:48:50.388708 (+ 109us) server_negotiation.cc:299] Negotiation successful
0621 07:48:50.388753 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 07:48:50.394130 12293 raft_consensus.cc:1261] T 2a689150b07446de8b12a54f69b7041f P c7d4911794b645378672dfe92ffb7ae1 [term 1 FOLLOWER]: Refusing update from remote peer eac82b8522434baa85a487191284b23c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:48:50.394142 12427 raft_consensus.cc:1261] T 2a689150b07446de8b12a54f69b7041f P aabe6ac224074bf2ad5e686897992b1e [term 1 FOLLOWER]: Refusing update from remote peer eac82b8522434baa85a487191284b23c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:48:50.394814 12500 consensus_queue.cc:1035] T 2a689150b07446de8b12a54f69b7041f P eac82b8522434baa85a487191284b23c [LEADER]: Connected to new peer: Peer: permanent_uuid: "aabe6ac224074bf2ad5e686897992b1e" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:48:50.395248 12512 consensus_queue.cc:1035] T 2a689150b07446de8b12a54f69b7041f P eac82b8522434baa85a487191284b23c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7d4911794b645378672dfe92ffb7ae1" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 33179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:48:50.402122 12517 mvcc.cc:205] Tried to move back new op lower bound from 7299187222087479296 to 7299187221754826752. Current Snapshot: MvccSnapshot[applied={T|T < 7299187222087479296}]
I20260621 07:48:50.403201 12516 mvcc.cc:205] Tried to move back new op lower bound from 7299187222087479296 to 7299187221754826752. Current Snapshot: MvccSnapshot[applied={T|T < 7299187222087479296}]
I20260621 07:48:50.409461 11987 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:48:50.409615 11987 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:48:50.411954 11987 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d2ad6971df1e4cc688fb208e70d83a14: Sending DeleteTablet for 3 replicas of tablet 2a689150b07446de8b12a54f69b7041f
I20260621 07:48:50.412712 12125 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a689150b07446de8b12a54f69b7041f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:48:50 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:39848
I20260621 07:48:50.412729 12407 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a689150b07446de8b12a54f69b7041f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:48:50 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:33384
I20260621 07:48:50.412729 12273 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a689150b07446de8b12a54f69b7041f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:48:50 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:46886
I20260621 07:48:50.413350 12524 tablet_replica.cc:330] T 2a689150b07446de8b12a54f69b7041f P eac82b8522434baa85a487191284b23c: stopping tablet replica
I20260621 07:48:50.413691 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 12070
I20260621 07:48:50.415450 12525 tablet_replica.cc:330] T 2a689150b07446de8b12a54f69b7041f P c7d4911794b645378672dfe92ffb7ae1: stopping tablet replica
I20260621 07:48:50.415709 12525 raft_consensus.cc:2229] T 2a689150b07446de8b12a54f69b7041f P c7d4911794b645378672dfe92ffb7ae1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:48:50.415977 12525 raft_consensus.cc:2258] T 2a689150b07446de8b12a54f69b7041f P c7d4911794b645378672dfe92ffb7ae1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:48:50.416270 12526 tablet_replica.cc:330] T 2a689150b07446de8b12a54f69b7041f P aabe6ac224074bf2ad5e686897992b1e: stopping tablet replica
I20260621 07:48:50.416502 12526 raft_consensus.cc:2229] T 2a689150b07446de8b12a54f69b7041f P aabe6ac224074bf2ad5e686897992b1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:48:50.416760 12526 raft_consensus.cc:2258] T 2a689150b07446de8b12a54f69b7041f P aabe6ac224074bf2ad5e686897992b1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:48:50.417820 12526 ts_tablet_manager.cc:1905] T 2a689150b07446de8b12a54f69b7041f P aabe6ac224074bf2ad5e686897992b1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:48:50.417945 12525 ts_tablet_manager.cc:1905] T 2a689150b07446de8b12a54f69b7041f P c7d4911794b645378672dfe92ffb7ae1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:48:50.419951 12526 ts_tablet_manager.cc:1918] T 2a689150b07446de8b12a54f69b7041f P aabe6ac224074bf2ad5e686897992b1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:48:50.420071 12526 log.cc:1196] T 2a689150b07446de8b12a54f69b7041f P aabe6ac224074bf2ad5e686897992b1e: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-2/wal/wals/2a689150b07446de8b12a54f69b7041f
I20260621 07:48:50.420387 12525 ts_tablet_manager.cc:1918] T 2a689150b07446de8b12a54f69b7041f P c7d4911794b645378672dfe92ffb7ae1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:48:50.420419 12526 ts_tablet_manager.cc:1939] T 2a689150b07446de8b12a54f69b7041f P aabe6ac224074bf2ad5e686897992b1e: Deleting consensus metadata
I20260621 07:48:50.420534 12525 log.cc:1196] T 2a689150b07446de8b12a54f69b7041f P c7d4911794b645378672dfe92ffb7ae1: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782028028522580-771-0/minicluster-data/ts-1/wal/wals/2a689150b07446de8b12a54f69b7041f
W20260621 07:48:50.420900 11975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.192.193:42937 (error 108)
I20260621 07:48:50.420936 12525 ts_tablet_manager.cc:1939] T 2a689150b07446de8b12a54f69b7041f P c7d4911794b645378672dfe92ffb7ae1: Deleting consensus metadata
I20260621 07:48:50.421111 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 12209
I20260621 07:48:50.422372 12487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:50.422095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:42937 (local address 127.0.0.1:39856)
0621 07:48:50.422210 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0621 07:48:50.422289 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.193:42937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260621 07:48:50.422343 11975 catalog_manager.cc:4841] TS aabe6ac224074bf2ad5e686897992b1e (127.0.192.195:43715): tablet 2a689150b07446de8b12a54f69b7041f (table test-table [id=45c9b13ca98a44ef88b107586754c1e6]) successfully deleted
W20260621 07:48:50.422536 11975 catalog_manager.cc:4584] TS eac82b8522434baa85a487191284b23c (127.0.192.193:42937): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2a689150b07446de8b12a54f69b7041f: Network error: Client connection negotiation failed: client connection to 127.0.192.193:42937: connect: Connection refused (error 111)
I20260621 07:48:50.426944 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 12344
I20260621 07:48:50.427682 12487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:48:50.427045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:33179 (local address 127.0.0.1:46894)
0621 07:48:50.427537 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0621 07:48:50.427596 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.194:33179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406}
I20260621 07:48:50.432101 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 11957
I20260621 07:48:50.438578 771 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260621 07:48:50.772624 771 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260621 07:48:50.772779 771 mini_ranger.cc:66] Stopping Ranger...
I20260621 07:48:51.117002 771 mini_ranger.cc:68] Stopped Ranger
2026-06-21 07:48:51.117 UTC [11542] LOG: received smart shutdown request
2026-06-21 07:48:51.120 UTC [11542] LOG: background worker "logical replication launcher" (PID 11551) exited with exit code 1
2026-06-21 07:48:51.120 UTC [11546] LOG: shutting down
2026-06-21 07:48:51.267 UTC [11542] LOG: database system is shut down
2026-06-21T07:48:51Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (42916 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:48:51 dist-test-slave-lzx3 krb5kdc[12536](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:48:51 dist-test-slave-lzx3 krb5kdc[12536](info): set up 2 sockets
Jun 21 07:48:51 dist-test-slave-lzx3 krb5kdc[12536](info): commencing operation
krb5kdc: starting...
W20260621 07:48:52.360121 771 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/test-user.keytab.
Jun 21 07:48:52 dist-test-slave-lzx3 krb5kdc[12536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:48:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:48:52Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.192.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.192.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.192.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/rangeradmin_127.0.192.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.192.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/rangeradmin_127.0.192.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.192.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.192.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.192.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/rangerlookup_127.0.192.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.192.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/rangerlookup_127.0.192.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.192.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.192.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.192.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/HTTP_127.0.192.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.192.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/HTTP_127.0.192.212@KRBTEST.COM.keytab.
I20260621 07:48:52.506891 771 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskeKA8lE/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/postgres -l logfile start
2026-06-21 07:48:54.752 UTC [12566] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-21 07:48:54.752 UTC [12566] LOG: listening on IPv4 address "127.0.192.212", port 49171
2026-06-21 07:48:54.756 UTC [12566] LOG: listening on Unix socket "/tmp/.s.PGSQL.49171"
2026-06-21 07:48:54.788 UTC [12569] LOG: database system was shut down at 2026-06-21 07:48:53 UTC
2026-06-21 07:48:54.792 UTC [12566] LOG: database system is ready to accept connections
I20260621 07:48:55.668130 771 mini_postgres.cc:96] Postgres bound to 49171
2026-06-21 07:48:55.674 UTC [12577] FATAL: database "slave" does not exist
127.0.192.212:49171 - accepting connections
I20260621 07:48:55.675189 771 mini_ranger.cc:161] Starting Ranger...
I20260621 07:48:55.693504 771 mini_ranger.cc:84] Created miniranger Postgres user
I20260621 07:48:56.126591 771 mini_ranger.cc:87] Created ranger Postgres database
I20260621 07:48:56.126681 771 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ranger-admin
2026-06-21 07:48:56,361 [I] DB FLAVOR :POSTGRES
2026-06-21 07:48:56,362 [I] --------- Verifying Ranger DB connection ---------
2026-06-21 07:48:56,362 [I] Checking connection..
2026-06-21 07:48:56,362 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:49171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-21 07:48:56,582 [I] Checking connection passed.
2026-06-21 07:48:56,582 [I] --------- Verifying version history table ---------
2026-06-21 07:48:56,582 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:49171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 07:48:56,814 [I] Table x_db_version_h does not exist in database ranger
2026-06-21 07:48:56,814 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:49171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 07:48:57,053 [I] Table x_db_version_h does not exist in database ranger
2026-06-21 07:48:57,054 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-21 07:48:57,054 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:49171/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-21 07:48:57.287 UTC [12641] WARNING: there is no transaction in progress
2026-06-21 07:48:57,292 [I] create_dbversion_catalog.sql file imported successfully
2026-06-21 07:48:57,293 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:49171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 07:48:57,506 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-21 07:48:57,507 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-21 07:48:57,507 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:49171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-21 07:48:57,754 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:49171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-21 07:48:57,970 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:49171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-lzx3.c.gcp-upstream.internal','N') ;"
2026-06-21 07:48:58,200 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:49171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-21 07:48:58,431 [I] Table x_portal_user does not exist in database ranger
2026-06-21 07:48:58,431 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:49171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-21 07:48:58,672 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-21 07:48:58,672 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:49171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-21 07:48:58,897 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:49171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-21 07:48:59,107 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-21 07:48:59,107 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:49171/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-21 07:48:59.948 UTC [12749] WARNING: there is no transaction in progress
2026-06-21 07:48:59.963 UTC [12749] WARNING: there is no transaction in progress
2026-06-21 07:48:59.978 UTC [12749] WARNING: there is no transaction in progress
2026-06-21 07:48:59.992 UTC [12749] WARNING: there is no transaction in progress
2026-06-21 07:49:00.008 UTC [12749] WARNING: there is no transaction in progress
2026-06-21 07:49:00.023 UTC [12749] WARNING: there is no transaction in progress
2026-06-21 07:49:00.070 UTC [12749] WARNING: there is no transaction in progress
2026-06-21 07:49:00.077 UTC [12749] WARNING: there is no transaction in progress
2026-06-21 07:49:00.090 UTC [12749] WARNING: there is no transaction in progress
2026-06-21 07:49:00.098 UTC [12749] WARNING: there is no transaction in progress
2026-06-21 07:49:00.107 UTC [12749] WARNING: there is no transaction in progress
2026-06-21 07:49:00.119 UTC [12749] WARNING: there is no transaction in progress
2026-06-21 07:49:00.127 UTC [12749] WARNING: there is no transaction in progress
2026-06-21 07:49:00.133 UTC [12749] WARNING: there is no transaction in progress
2026-06-21 07:49:00.139 UTC [12749] WARNING: there is no transaction in progress
2026-06-21 07:49:00.145 UTC [12749] WARNING: there is no transaction in progress
2026-06-21 07:49:00,707 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-21 07:49:00,708 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:49171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-21 07:49:00,941 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-21 07:49:00,942 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:49171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-21 07:49:01,167 [I] Table x_portal_user already exists in database 'ranger'
2026-06-21 07:49:01,167 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:49171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-21 07:49:01,406 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-21 07:49:01,407 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:49171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-21 07:49:01,629 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:49171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-21 07:49:01,839 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:49171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-lzx3.c.gcp-upstream.internal';"
2026-06-21 07:49:02,038 [I] CORE_DB_SCHEMA import status has been updated
2026-06-21 07:49:02,038 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:49171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-21 07:49:02,273 [I] DB_PATCHES have already been applied
I20260621 07:49:02.278262 771 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ranger-admin:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskeKA8lE/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260621 07:49:02.278326 771 mini_ranger.cc:193] Using host: 127.0.192.212
I20260621 07:49:02.281601 771 mini_ranger.cc:227] Ranger admin URL: http://127.0.192.212:57055
Jun 21, 2026 7:49:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 21, 2026 7:49:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 21, 2026 7:49:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ranger-admin/ews/webapp
Jun 21, 2026 7:49:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 21, 2026 7:49:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 21, 2026 7:49:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ranger-admin/ews/webapp].
Jun 21, 2026 7:49:03 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260621 07:49:03.331214 771 mini_ranger.cc:160] Time spent starting Ranger: real 7.656s user 0.000s sys 0.009s
Jun 21, 2026 7:49:20 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 21, 2026 7:49:20 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 21, 2026 7:49:20 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 21, 2026 7:49:20 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 21, 2026 7:49:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 21, 2026 7:49:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 21, 2026 7:49:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 21, 2026 7:49:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 21, 2026 7:49:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 21, 2026 7:49:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 21, 2026 7:49:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 21, 2026 7:49:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 21, 2026 7:49:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 21, 2026 7:49:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 21, 2026 7:49:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 21, 2026 7:49:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 21, 2026 7:49:20 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 21, 2026 7:49:21 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260621 07:49:22.018242 771 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.192.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.192.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.192.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/rangerkms_127.0.192.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.192.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/rangerkms_127.0.192.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.192.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.192.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.192.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/HTTP_127.0.192.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.192.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/HTTP_127.0.192.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 21 07:49:22 dist-test-slave-lzx3 krb5kdc[12536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028162, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260621 07:49:22.102826 771 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260621 07:49:22.112496 771 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260621 07:49:22.896625 771 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260621 07:49:22.896704 771 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ranger-kms
I20260621 07:49:22.896724 771 mini_ranger_kms.cc:227] Using postgres at 127.0.192.212:49171
2026-06-21 07:49:23,056 [I] DB FLAVOR :POSTGRES
2026-06-21 07:49:23,056 [I] --------- Verifying Ranger DB connection ---------
2026-06-21 07:49:23,056 [I] Checking connection
2026-06-21 07:49:23,057 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:49171/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-21 07:49:23,269 [I] connection success
2026-06-21 07:49:23,269 [I] --------- Verifying Ranger DB tables ---------
2026-06-21 07:49:23,269 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-21 07:49:23,270 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:49171/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-21 07:49:23,484 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-21 07:49:23,484 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-21 07:49:23,484 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-21 07:49:23,484 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.192.212:49171/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-21 07:49:23,725 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260621 07:49:24.135066 771 mini_ranger_kms.cc:310] Created kms service
I20260621 07:49:24.329591 771 mini_ranger_kms.cc:326] Created kudu user
I20260621 07:49:24.368979 771 mini_ranger_kms.cc:343] Created rangerkms user
Jun 21 07:49:24 dist-test-slave-lzx3 krb5kdc[12536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028162, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.192.212@KRBTEST.COM
I20260621 07:49:24.711711 771 mini_ranger_kms.cc:384] Added ranger policy
I20260621 07:49:24.711863 771 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ranger-kms:/tmp/dist-test-taskeKA8lE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskeKA8lE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260621 07:49:24.711889 771 mini_ranger_kms.cc:240] Using host: 127.0.192.212
I20260621 07:49:24.715045 771 mini_ranger_kms.cc:276] Ranger KMS PID: 12929
I20260621 07:49:24.715128 771 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.192.212:57955
Jun 21, 2026 7:49:25 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 21, 2026 7:49:25 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 21, 2026 7:49:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 21, 2026 7:49:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 21, 2026 7:49:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ranger-kms/ews/webapp].
Jun 21, 2026 7:49:26 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 21 07:49:26 dist-test-slave-lzx3 krb5kdc[12536](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782028166, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.192.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:49:26.830627 771 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.728s user 0.007s sys 0.137s
I20260621 07:49:26.830719 771 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260621 07:49:26.830787 771 mini_ranger_kms.cc:401] 127.0.192.212:57955/kms/v1/keys
Jun 21 07:49:27 dist-test-slave-lzx3 krb5kdc[12536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028166, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.192.212@KRBTEST.COM for HTTP/127.0.192.212@KRBTEST.COM
Jun 21 07:49:27 dist-test-slave-lzx3 krb5kdc[12536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028166, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.192.212@KRBTEST.COM for HTTP/127.0.192.212@KRBTEST.COM
Jun 21 07:49:29 dist-test-slave-lzx3 krb5kdc[12536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.192.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
I20260621 07:49:29.514820 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:39807
--webserver_interface=127.0.192.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:43077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:39807
--ranger_config_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.192.212:57955/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:49:29.619586 12979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:49:29.619807 12979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:49:29.619843 12979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:49:29.623728 12979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:49:29.623776 12979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:49:29.623793 12979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:49:29.623808 12979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:49:29.623821 12979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:49:29.629384 12979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:43077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.192.212:57955/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:39807
--ranger_config_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:39807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:49:29.630249 12979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:49:29.631170 12979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:49:29.639097 12985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:49:29.639097 12988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:49:29.639109 12986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:49:29.639528 12979 server_base.cc:1034] running on GCE node
I20260621 07:49:29.640069 12979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:49:29.641222 12979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:49:29.642400 12979 hybrid_clock.cc:648] HybridClock initialized: now 1782028169642373 us; error 44 us; skew 500 ppm
Jun 21 07:49:29 dist-test-slave-lzx3 krb5kdc[12536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:49:29.644975 12979 init.cc:377] Logged in from keytab as kudu/127.0.192.254@KRBTEST.COM (short username kudu)
I20260621 07:49:29.645717 12979 webserver.cc:460] Webserver started at http://127.0.192.254:43785/ using document root <none> and password file <none>
I20260621 07:49:29.646185 12979 fs_manager.cc:359] Metadata directory not provided
I20260621 07:49:29.646257 12979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:49:29.646441 12979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 07:49:29 dist-test-slave-lzx3 krb5kdc[12536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for HTTP/127.0.192.212@KRBTEST.COM
I20260621 07:49:29.703572 12979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "3990f8b0921a42c3ac2abaf0b7287be9"
format_stamp: "Formatted at 2026-06-21 07:49:29 on dist-test-slave-lzx3"
server_key: "c6f6a2c122c2773bb323618cc846bf52"
server_key_iv: "478a8b395e2eea5c2a98bb6bd721682d"
server_key_version: "kuduclusterkey@0"
I20260621 07:49:29.704260 12979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "3990f8b0921a42c3ac2abaf0b7287be9"
format_stamp: "Formatted at 2026-06-21 07:49:29 on dist-test-slave-lzx3"
server_key: "c6f6a2c122c2773bb323618cc846bf52"
server_key_iv: "478a8b395e2eea5c2a98bb6bd721682d"
server_key_version: "kuduclusterkey@0"
I20260621 07:49:29.709262 12979 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 07:49:29.725337 12995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:29.726768 12979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 07:49:29.726969 12979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "3990f8b0921a42c3ac2abaf0b7287be9"
format_stamp: "Formatted at 2026-06-21 07:49:29 on dist-test-slave-lzx3"
server_key: "c6f6a2c122c2773bb323618cc846bf52"
server_key_iv: "478a8b395e2eea5c2a98bb6bd721682d"
server_key_version: "kuduclusterkey@0"
I20260621 07:49:29.727283 12979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:49:29.743211 12979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:49:29.746297 12979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:49:29.746563 12979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:49:29.754468 12979 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:39807
I20260621 07:49:29.754503 13047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:39807 every 8 connection(s)
I20260621 07:49:29.755323 12979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:49:29.757876 13048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:49:29.761299 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 12979
I20260621 07:49:29.761408 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:49:29.761687 771 external_mini_cluster.cc:1428] Setting key ecdc88eb08e85d1199094ba6e26c9578
I20260621 07:49:29.764343 13048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9: Bootstrap starting.
I20260621 07:49:29.766883 13048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9: Neither blocks nor log segments found. Creating new log.
I20260621 07:49:29.767777 13048 log.cc:824] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9: Log is configured to *not* fsync() on all Append() calls
I20260621 07:49:29.771427 13048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9: No bootstrap required, opened a new log
Jun 21 07:49:29 dist-test-slave-lzx3 krb5kdc[12536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:49:29.774971 13048 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3990f8b0921a42c3ac2abaf0b7287be9" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 39807 } }
I20260621 07:49:29.775252 13048 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:49:29.775323 13048 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3990f8b0921a42c3ac2abaf0b7287be9, State: Initialized, Role: FOLLOWER
I20260621 07:49:29.775806 13048 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3990f8b0921a42c3ac2abaf0b7287be9" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 39807 } }
I20260621 07:49:29.775964 13048 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:49:29.776067 13048 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:49:29.776177 13048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:49:29.777184 13048 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3990f8b0921a42c3ac2abaf0b7287be9" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 39807 } }
I20260621 07:49:29.777505 13048 leader_election.cc:302] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3990f8b0921a42c3ac2abaf0b7287be9; no voters:
I20260621 07:49:29.777748 13051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:29.763075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60514 (local address 127.0.192.254:39807)
0621 07:49:29.763656 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:29.763674 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:29.763739 (+ 65us) server_negotiation.cc:407] Connection header received
0621 07:49:29.764548 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:29.764595 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:29.765009 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:29.765344 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:49:29.766446 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:29.767496 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:29.768487 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:29.768967 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:29.769147 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:29.772600 (+ 3453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:49:29.772610 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:49:29.772627 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:49:29.772655 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:49:29.774445 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:29.774940 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:29.774946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:29.774951 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:29.775109 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:29.775473 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:29.775477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:29.775478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:29.775961 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:49:29.776156 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:29.776575 (+ 419us) server_negotiation.cc:299] Negotiation successful
0621 07:49:29.776821 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":112,"threads_started":1}
I20260621 07:49:29.777804 13048 leader_election.cc:288] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:49:29.778050 13053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:49:29.778285 13053 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9 [term 1 LEADER]: Becoming Leader. State: Replica: 3990f8b0921a42c3ac2abaf0b7287be9, State: Running, Role: LEADER
I20260621 07:49:29.778604 13053 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3990f8b0921a42c3ac2abaf0b7287be9" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 39807 } }
I20260621 07:49:29.780061 13053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3990f8b0921a42c3ac2abaf0b7287be9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3990f8b0921a42c3ac2abaf0b7287be9" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 39807 } } }
I20260621 07:49:29.780244 13053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9 [sys.catalog]: This master's current role is: LEADER
I20260621 07:49:29.780432 13054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3990f8b0921a42c3ac2abaf0b7287be9. Latest consensus state: current_term: 1 leader_uuid: "3990f8b0921a42c3ac2abaf0b7287be9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3990f8b0921a42c3ac2abaf0b7287be9" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 39807 } } }
I20260621 07:49:29.780599 13054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9 [sys.catalog]: This master's current role is: LEADER
I20260621 07:49:29.781102 13048 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:49:29.781255 13057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:49:29.785753 13057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:49:29.786109 13048 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260621 07:49:29.799599 13057 catalog_manager.cc:1347] Generated new cluster ID: d942ca7ff8c3482da660f5fbff0bc61c
I20260621 07:49:29.799674 13057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:49:29.844131 13057 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:49:29.845466 13057 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:49:29.851811 13057 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9: Generated new TSK 0
I20260621 07:49:29.852711 13057 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.192.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab.
Entry for principal kudu/127.0.192.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab.
I20260621 07:49:30.481113 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.192.212:57955/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.193:0
--local_ip_for_outbound_sockets=127.0.192.193
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:39807
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:43077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:49:30.609536 13093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:49:30.609782 13093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:49:30.609825 13093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:49:30.613767 13093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:49:30.613847 13093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:49:30.613950 13093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.193
I20260621 07:49:30.619252 13093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:43077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.192.212:57955/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:39807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.193
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:49:30.620258 13093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:49:30.621261 13093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:49:30.631093 13099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:49:30.631742 13098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:49:30.632011 13101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:49:30.632246 13093 server_base.cc:1034] running on GCE node
I20260621 07:49:30.632632 13093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:49:30.633212 13093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:49:30.636518 13093 hybrid_clock.cc:648] HybridClock initialized: now 1782028170636440 us; error 83 us; skew 500 ppm
Jun 21 07:49:30 dist-test-slave-lzx3 krb5kdc[12536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:49:30.643565 13093 init.cc:377] Logged in from keytab as kudu/127.0.192.193@KRBTEST.COM (short username kudu)
I20260621 07:49:30.644419 13093 webserver.cc:460] Webserver started at http://127.0.192.193:36339/ using document root <none> and password file <none>
I20260621 07:49:30.644974 13093 fs_manager.cc:359] Metadata directory not provided
I20260621 07:49:30.645062 13093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:49:30.645295 13093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 07:49:30 dist-test-slave-lzx3 krb5kdc[12536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for HTTP/127.0.192.212@KRBTEST.COM
I20260621 07:49:30.658779 13093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-0/data/instance:
uuid: "0e5e14b71ac349ea8747b3dcb01553af"
format_stamp: "Formatted at 2026-06-21 07:49:30 on dist-test-slave-lzx3"
server_key: "96335362c6c9de64bbff0053e8d15ca2"
server_key_iv: "6c93dfadd45fd976dc460196b1738f6d"
server_key_version: "kuduclusterkey@0"
I20260621 07:49:30.659420 13093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-0/wal/instance:
uuid: "0e5e14b71ac349ea8747b3dcb01553af"
format_stamp: "Formatted at 2026-06-21 07:49:30 on dist-test-slave-lzx3"
server_key: "96335362c6c9de64bbff0053e8d15ca2"
server_key_iv: "6c93dfadd45fd976dc460196b1738f6d"
server_key_version: "kuduclusterkey@0"
I20260621 07:49:30.664017 13093 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260621 07:49:30.684151 13110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:30.685711 13093 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260621 07:49:30.685899 13093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-0/wal
uuid: "0e5e14b71ac349ea8747b3dcb01553af"
format_stamp: "Formatted at 2026-06-21 07:49:30 on dist-test-slave-lzx3"
server_key: "96335362c6c9de64bbff0053e8d15ca2"
server_key_iv: "6c93dfadd45fd976dc460196b1738f6d"
server_key_version: "kuduclusterkey@0"
I20260621 07:49:30.686098 13093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:49:30.703243 13093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:49:30.706557 13093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:49:30.706815 13093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:49:30.707548 13093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:49:30.708655 13093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:49:30.708770 13093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:30.708901 13093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:49:30.708986 13093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:30.722369 13093 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.193:38271
I20260621 07:49:30.723408 13093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
I20260621 07:49:30.728790 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 13093
I20260621 07:49:30.728946 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-0/wal/instance
I20260621 07:49:30.729234 771 external_mini_cluster.cc:1428] Setting key bc197948ece3f44e91d52a79c2fb7688
I20260621 07:49:30.725003 13223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.193:38271 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.192.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.194@KRBTEST.COM" created.
Jun 21 07:49:30 dist-test-slave-lzx3 krb5kdc[12536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:49:30.754998 13226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:30.726880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:39807 (local address 127.0.192.193:58565)
0621 07:49:30.727505 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:30.727560 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0621 07:49:30.736925 (+ 9365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:49:30.737875 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:49:30.737884 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:49:30.738208 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:49:30.738851 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:30.738870 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:30.740031 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:30.740035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:30.741449 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:30.741459 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:30.742309 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:30.742314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:30.742615 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:30.743365 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:49:30.743385 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:49:30.745163 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:49:30.747596 (+ 2433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:30.747602 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:30.747618 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:30.747952 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:30.748285 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:30.748289 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:30.748291 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:30.748457 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:30.753172 (+ 4715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:49:30.753179 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:49:30.753517 (+ 338us) client_negotiation.cc:769] Sending connection context
0621 07:49:30.753703 (+ 186us) client_negotiation.cc:240] Negotiation successful
0621 07:49:30.754019 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":151,"threads_started":1}
I20260621 07:49:30.755852 13227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:30.727994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:58565 (local address 127.0.192.254:39807)
0621 07:49:30.737364 (+ 9370us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:30.737372 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:30.737385 (+ 13us) server_negotiation.cc:407] Connection header received
0621 07:49:30.737423 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:30.737426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:30.737469 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:30.737566 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:49:30.739009 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:30.739892 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:30.741705 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:30.742036 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:30.742127 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:30.745376 (+ 3249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:49:30.745381 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:49:30.745384 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:49:30.745411 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:49:30.747456 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:30.748078 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:30.748082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:30.748085 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:30.748178 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:30.752805 (+ 4627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:30.752809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:30.752811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:30.753031 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:49:30.755495 (+ 2464us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:30.755572 (+ 77us) server_negotiation.cc:299] Negotiation successful
0621 07:49:30.755693 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9297,"thread_start_us":94,"threads_started":1}
I20260621 07:49:30.762276 13224 heartbeater.cc:347] Connected to a master server at 127.0.192.254:39807
I20260621 07:49:30.762518 13224 heartbeater.cc:464] Registering TS with master...
I20260621 07:49:30.763096 13224 heartbeater.cc:511] Master 127.0.192.254:39807 requested a full tablet report, sending...
I20260621 07:49:30.764899 13012 ts_manager.cc:194] Registered new tserver with Master: 0e5e14b71ac349ea8747b3dcb01553af (127.0.192.193:38271)
I20260621 07:49:30.766310 13012 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.193@KRBTEST.COM'} at 127.0.192.193:58565
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab.
Entry for principal kudu/127.0.192.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab.
I20260621 07:49:30.784006 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.192.212:57955/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.194:0
--local_ip_for_outbound_sockets=127.0.192.194
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:39807
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:43077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 07:49:30 dist-test-slave-lzx3 krb5kdc[12536](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782028170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260621 07:49:30.932787 13232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:49:30.933228 13232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:49:30.933328 13232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:49:30.938503 13232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:49:30.938616 13232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:49:30.938730 13232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.194
I20260621 07:49:30.945788 13232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:43077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.192.212:57955/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:39807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.194
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:49:30.947090 13232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:49:30.948330 13232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 07:49:30.960071 13232 server_base.cc:1034] running on GCE node
W20260621 07:49:30.960779 13239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:49:30.960960 13241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:49:30.960807 13238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:49:30.961418 13232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:49:30.962018 13232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:49:30.963188 13232 hybrid_clock.cc:648] HybridClock initialized: now 1782028170963171 us; error 32 us; skew 500 ppm
Jun 21 07:49:30 dist-test-slave-lzx3 krb5kdc[12536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:49:30.966361 13232 init.cc:377] Logged in from keytab as kudu/127.0.192.194@KRBTEST.COM (short username kudu)
I20260621 07:49:30.967301 13232 webserver.cc:460] Webserver started at http://127.0.192.194:42833/ using document root <none> and password file <none>
I20260621 07:49:30.967945 13232 fs_manager.cc:359] Metadata directory not provided
I20260621 07:49:30.968019 13232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:49:30.968259 13232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 07:49:30 dist-test-slave-lzx3 krb5kdc[12536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for HTTP/127.0.192.212@KRBTEST.COM
I20260621 07:49:30.987856 13232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-1/data/instance:
uuid: "ff83fda783ad4c188b9bb8427d96de45"
format_stamp: "Formatted at 2026-06-21 07:49:30 on dist-test-slave-lzx3"
server_key: "bcadaa1e76a6edc6e54020932c60011c"
server_key_iv: "f01785852453e182f4c0404ec32f50fe"
server_key_version: "kuduclusterkey@0"
I20260621 07:49:30.988690 13232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-1/wal/instance:
uuid: "ff83fda783ad4c188b9bb8427d96de45"
format_stamp: "Formatted at 2026-06-21 07:49:30 on dist-test-slave-lzx3"
server_key: "bcadaa1e76a6edc6e54020932c60011c"
server_key_iv: "f01785852453e182f4c0404ec32f50fe"
server_key_version: "kuduclusterkey@0"
I20260621 07:49:30.994134 13232 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260621 07:49:31.010306 13248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:31.011888 13232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:49:31.011991 13232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-1/wal
uuid: "ff83fda783ad4c188b9bb8427d96de45"
format_stamp: "Formatted at 2026-06-21 07:49:30 on dist-test-slave-lzx3"
server_key: "bcadaa1e76a6edc6e54020932c60011c"
server_key_iv: "f01785852453e182f4c0404ec32f50fe"
server_key_version: "kuduclusterkey@0"
I20260621 07:49:31.012090 13232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:49:31.028017 13232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:49:31.031358 13232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:49:31.031601 13232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:49:31.032241 13232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:49:31.033286 13232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:49:31.033375 13232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:31.033457 13232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:49:31.033510 13232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:31.045463 13232 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.194:33313
I20260621 07:49:31.046442 13232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
I20260621 07:49:31.051615 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 13232
I20260621 07:49:31.051740 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-1/wal/instance
I20260621 07:49:31.052043 771 external_mini_cluster.cc:1428] Setting key 968780345c8cc7eccf6a0ab9064a2b36
I20260621 07:49:31.047654 13361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.194:33313 every 8 connection(s)
Jun 21 07:49:31 dist-test-slave-lzx3 krb5kdc[12536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:49:31.069814 13227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:31.052460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:37673 (local address 127.0.192.254:39807)
0621 07:49:31.052612 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:31.052617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:31.057042 (+ 4425us) server_negotiation.cc:407] Connection header received
0621 07:49:31.058117 (+ 1075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:31.058121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:31.058189 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:31.058288 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:49:31.059857 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:31.060688 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:31.062241 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:31.062573 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:31.062652 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:31.065623 (+ 2971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:49:31.065628 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:49:31.065630 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:49:31.065656 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:49:31.067401 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:31.068019 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:31.068022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:31.068024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:31.068111 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:31.068544 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:31.068547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:31.068549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:31.068797 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:49:31.068868 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:31.069488 (+ 620us) server_negotiation.cc:299] Negotiation successful
0621 07:49:31.069634 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260621 07:49:31.071758 13364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:31.050422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:39807 (local address 127.0.192.194:37673)
0621 07:49:31.056876 (+ 6454us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:31.056934 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0621 07:49:31.057885 (+ 951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:49:31.058614 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:49:31.058624 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:49:31.059004 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:49:31.059667 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:31.059689 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:31.060847 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:31.060850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:31.062114 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:31.062125 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:31.062863 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:31.062867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:31.063059 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:31.063696 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:49:31.063727 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:49:31.065438 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:49:31.067532 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:31.067538 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:31.067552 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:31.067895 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:31.068262 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:31.068265 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:31.068267 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:31.068430 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:31.068948 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:49:31.068953 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:49:31.069223 (+ 270us) client_negotiation.cc:769] Sending connection context
0621 07:49:31.070476 (+ 1253us) client_negotiation.cc:240] Negotiation successful
0621 07:49:31.070807 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6177,"thread_start_us":135,"threads_started":1}
I20260621 07:49:31.073143 13362 heartbeater.cc:347] Connected to a master server at 127.0.192.254:39807
I20260621 07:49:31.073387 13362 heartbeater.cc:464] Registering TS with master...
I20260621 07:49:31.073987 13362 heartbeater.cc:511] Master 127.0.192.254:39807 requested a full tablet report, sending...
I20260621 07:49:31.075127 13012 ts_manager.cc:194] Registered new tserver with Master: ff83fda783ad4c188b9bb8427d96de45 (127.0.192.194:33313)
I20260621 07:49:31.075865 13012 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.194@KRBTEST.COM'} at 127.0.192.194:37673
WARNING: no policy specified for kudu/127.0.192.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab.
Entry for principal kudu/127.0.192.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab.
I20260621 07:49:31.106874 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.192.212:57955/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.195:0
--local_ip_for_outbound_sockets=127.0.192.195
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:39807
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:43077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:49:31.255218 13367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:49:31.255513 13367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:49:31.255561 13367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:49:31.261018 13367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:49:31.261077 13367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:49:31.261150 13367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.195
I20260621 07:49:31.267807 13367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:43077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.192.212:57955/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:39807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.195
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:49:31.268905 13367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:49:31.270035 13367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:49:31.281342 13372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:49:31.281708 13375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:49:31.282507 13367 server_base.cc:1034] running on GCE node
W20260621 07:49:31.282729 13373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:49:31.283130 13367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:49:31.283820 13367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:49:31.285044 13367 hybrid_clock.cc:648] HybridClock initialized: now 1782028171284969 us; error 69 us; skew 500 ppm
Jun 21 07:49:31 dist-test-slave-lzx3 krb5kdc[12536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:49:31.288515 13367 init.cc:377] Logged in from keytab as kudu/127.0.192.195@KRBTEST.COM (short username kudu)
I20260621 07:49:31.289403 13367 webserver.cc:460] Webserver started at http://127.0.192.195:37697/ using document root <none> and password file <none>
I20260621 07:49:31.290045 13367 fs_manager.cc:359] Metadata directory not provided
I20260621 07:49:31.290126 13367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:49:31.290336 13367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 07:49:31 dist-test-slave-lzx3 krb5kdc[12536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for HTTP/127.0.192.212@KRBTEST.COM
I20260621 07:49:31.323328 13367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-2/data/instance:
uuid: "c459bfb8155f4289a67d84e9f19dce87"
format_stamp: "Formatted at 2026-06-21 07:49:31 on dist-test-slave-lzx3"
server_key: "ed8f817c1b4c26c3762daee15bacd808"
server_key_iv: "2628b956baa871e38a546e11e0869a86"
server_key_version: "kuduclusterkey@0"
I20260621 07:49:31.324184 13367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-2/wal/instance:
uuid: "c459bfb8155f4289a67d84e9f19dce87"
format_stamp: "Formatted at 2026-06-21 07:49:31 on dist-test-slave-lzx3"
server_key: "ed8f817c1b4c26c3762daee15bacd808"
server_key_iv: "2628b956baa871e38a546e11e0869a86"
server_key_version: "kuduclusterkey@0"
I20260621 07:49:31.329732 13367 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
Jun 21 07:49:31 dist-test-slave-lzx3 krb5kdc[12536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for HTTP/127.0.192.212@KRBTEST.COM
I20260621 07:49:31.358172 13382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:31.359791 13367 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.003s sys 0.000s
I20260621 07:49:31.359927 13367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-2/wal
uuid: "c459bfb8155f4289a67d84e9f19dce87"
format_stamp: "Formatted at 2026-06-21 07:49:31 on dist-test-slave-lzx3"
server_key: "ed8f817c1b4c26c3762daee15bacd808"
server_key_iv: "2628b956baa871e38a546e11e0869a86"
server_key_version: "kuduclusterkey@0"
I20260621 07:49:31.360070 13367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:49:31.479425 13367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:49:31.492316 13367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:49:31.492492 13367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:49:31.493155 13367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:49:31.494160 13367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:49:31.494235 13367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:31.494293 13367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:49:31.494340 13367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:31.504027 13367 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.195:46451
I20260621 07:49:31.513245 13367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
I20260621 07:49:31.517393 13495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.195:46451 every 8 connection(s)
I20260621 07:49:31.524992 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 13367
I20260621 07:49:31.525110 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-2/wal/instance
I20260621 07:49:31.525434 771 external_mini_cluster.cc:1428] Setting key c7a5ab5631660ce95c0784cb7186f222
Jun 21 07:49:31 dist-test-slave-lzx3 krb5kdc[12536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:49:31.544166 13227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:31.525833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:34735 (local address 127.0.192.254:39807)
0621 07:49:31.526015 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:31.526020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:31.529695 (+ 3675us) server_negotiation.cc:407] Connection header received
0621 07:49:31.533285 (+ 3590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:31.533289 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:31.533342 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:31.533399 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:49:31.534981 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:31.535739 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:31.537317 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:31.537639 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:31.537697 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:31.540464 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:49:31.540468 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:49:31.540471 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:49:31.540496 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:49:31.542054 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:31.542657 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:31.542660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:31.542662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:31.542738 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:31.543087 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:31.543090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:31.543092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:31.543300 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:49:31.543362 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:31.543924 (+ 562us) server_negotiation.cc:299] Negotiation successful
0621 07:49:31.544026 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260621 07:49:31.545470 13498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:31.522615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:39807 (local address 127.0.192.195:34735)
0621 07:49:31.529493 (+ 6878us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:31.529575 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0621 07:49:31.532971 (+ 3396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:49:31.533796 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:49:31.533804 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:49:31.534145 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:49:31.534803 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:31.534824 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:31.535880 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:31.535883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:31.537165 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:31.537174 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:31.537875 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:31.537878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:31.538040 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:31.538689 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:49:31.538707 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:49:31.540292 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:49:31.542177 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:31.542182 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:31.542194 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:31.542529 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:31.542839 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:31.542842 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:31.542843 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:31.543000 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:31.543436 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:49:31.543441 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:49:31.543690 (+ 249us) client_negotiation.cc:769] Sending connection context
0621 07:49:31.544301 (+ 611us) client_negotiation.cc:240] Negotiation successful
0621 07:49:31.544604 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6654,"thread_start_us":106,"threads_started":1}
I20260621 07:49:31.546680 13496 heartbeater.cc:347] Connected to a master server at 127.0.192.254:39807
I20260621 07:49:31.546883 13496 heartbeater.cc:464] Registering TS with master...
I20260621 07:49:31.547408 13496 heartbeater.cc:511] Master 127.0.192.254:39807 requested a full tablet report, sending...
I20260621 07:49:31.548370 13012 ts_manager.cc:194] Registered new tserver with Master: c459bfb8155f4289a67d84e9f19dce87 (127.0.192.195:46451)
I20260621 07:49:31.548964 13012 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.195@KRBTEST.COM'} at 127.0.192.195:34735
I20260621 07:49:31.551721 771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:49:31.552378 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 13093
I20260621 07:49:31.559839 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 13232
I20260621 07:49:31.564970 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 13367
I20260621 07:49:31.572530 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 12979
I20260621 07:49:31.577427 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:39807
--webserver_interface=127.0.192.254
--webserver_port=43785
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:43077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:39807
--ranger_config_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.192.212:57955/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.192.212:57955/kms with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:49:31.683739 13499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:49:31.683985 13499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:49:31.684063 13499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:49:31.687738 13499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:49:31.687824 13499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:49:31.687878 13499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:49:31.687942 13499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:49:31.687981 13499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:49:31.693193 13499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:43077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.192.212:57955/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:39807
--ranger_config_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:39807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=43785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:49:31.694095 13499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:49:31.695019 13499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:49:31.703775 13504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:49:31.703795 13507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:49:31.704062 13499 server_base.cc:1034] running on GCE node
W20260621 07:49:31.703876 13505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:49:31.704846 13499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:49:31.705986 13499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:49:31.707173 13499 hybrid_clock.cc:648] HybridClock initialized: now 1782028171707141 us; error 43 us; skew 500 ppm
Jun 21 07:49:31 dist-test-slave-lzx3 krb5kdc[12536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:49:31.710318 13499 init.cc:377] Logged in from keytab as kudu/127.0.192.254@KRBTEST.COM (short username kudu)
I20260621 07:49:31.711109 13499 webserver.cc:460] Webserver started at http://127.0.192.254:43785/ using document root <none> and password file <none>
I20260621 07:49:31.711568 13499 fs_manager.cc:359] Metadata directory not provided
I20260621 07:49:31.711624 13499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 21 07:49:31 dist-test-slave-lzx3 krb5kdc[12536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for HTTP/127.0.192.212@KRBTEST.COM
I20260621 07:49:31.755023 13499 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260621 07:49:31.757071 13515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:31.758279 13499 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:49:31.758388 13499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "3990f8b0921a42c3ac2abaf0b7287be9"
format_stamp: "Formatted at 2026-06-21 07:49:29 on dist-test-slave-lzx3"
server_key: "c6f6a2c122c2773bb323618cc846bf52"
server_key_iv: "478a8b395e2eea5c2a98bb6bd721682d"
server_key_version: "kuduclusterkey@0"
I20260621 07:49:31.758852 13499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:49:31.776651 13499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:49:31.778997 13499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:49:31.779167 13499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:49:31.786702 13499 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:39807
I20260621 07:49:31.786711 13567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:39807 every 8 connection(s)
I20260621 07:49:31.787696 13499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:49:31.791622 13568 sys_catalog.cc:263] Verifying existing consensus state
I20260621 07:49:31.793172 13568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9: Bootstrap starting.
I20260621 07:49:31.793517 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 13499
I20260621 07:49:31.793956 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.192.212:57955/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.193:38271
--local_ip_for_outbound_sockets=127.0.192.193
--tserver_master_addrs=127.0.192.254:39807
--webserver_port=36339
--webserver_interface=127.0.192.193
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:43077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.192.212:57955/kms with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:49:31.814653 13568 log.cc:824] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9: Log is configured to *not* fsync() on all Append() calls
I20260621 07:49:31.821971 13568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 07:49:31.822330 13568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9: Bootstrap complete.
I20260621 07:49:31.827275 13568 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3990f8b0921a42c3ac2abaf0b7287be9" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 39807 } }
I20260621 07:49:31.827765 13568 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3990f8b0921a42c3ac2abaf0b7287be9, State: Initialized, Role: FOLLOWER
I20260621 07:49:31.828310 13568 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3990f8b0921a42c3ac2abaf0b7287be9" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 39807 } }
I20260621 07:49:31.828440 13568 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:49:31.828521 13568 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:49:31.828634 13568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9 [term 1 FOLLOWER]: Advancing to term 2
I20260621 07:49:31.830798 13568 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3990f8b0921a42c3ac2abaf0b7287be9" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 39807 } }
I20260621 07:49:31.831133 13568 leader_election.cc:302] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3990f8b0921a42c3ac2abaf0b7287be9; no voters:
I20260621 07:49:31.831449 13568 leader_election.cc:288] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260621 07:49:31.831589 13572 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9 [term 2 FOLLOWER]: Leader election won for term 2
I20260621 07:49:31.831863 13572 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9 [term 2 LEADER]: Becoming Leader. State: Replica: 3990f8b0921a42c3ac2abaf0b7287be9, State: Running, Role: LEADER
I20260621 07:49:31.832178 13572 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3990f8b0921a42c3ac2abaf0b7287be9" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 39807 } }
I20260621 07:49:31.832547 13568 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:49:31.833181 13572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3990f8b0921a42c3ac2abaf0b7287be9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3990f8b0921a42c3ac2abaf0b7287be9" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 39807 } } }
I20260621 07:49:31.833338 13572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9 [sys.catalog]: This master's current role is: LEADER
I20260621 07:49:31.833549 13572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3990f8b0921a42c3ac2abaf0b7287be9. Latest consensus state: current_term: 2 leader_uuid: "3990f8b0921a42c3ac2abaf0b7287be9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3990f8b0921a42c3ac2abaf0b7287be9" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 39807 } } }
I20260621 07:49:31.833662 13572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9 [sys.catalog]: This master's current role is: LEADER
I20260621 07:49:31.833920 13580 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:49:31.837045 13580 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:49:31.837522 13568 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260621 07:49:31.837787 13580 catalog_manager.cc:1259] Loaded cluster ID: d942ca7ff8c3482da660f5fbff0bc61c
I20260621 07:49:31.837960 13580 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:49:31.840360 13580 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:49:31.841087 13580 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9: Loaded TSK: 0
I20260621 07:49:31.842232 13580 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 07:49:31.943022 13570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:49:31.943455 13570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:49:31.943588 13570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:49:31.949399 13570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:49:31.949533 13570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:49:31.949709 13570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.193
I20260621 07:49:31.956872 13570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:43077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.192.212:57955/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.193:38271
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.192.193
--webserver_port=36339
--tserver_master_addrs=127.0.192.254:39807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.193
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:49:31.958138 13570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:49:31.959357 13570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:49:31.971586 13605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:49:31.971735 13608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:49:31.972116 13570 server_base.cc:1034] running on GCE node
W20260621 07:49:31.972708 13606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:49:31.973165 13570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:49:31.973868 13570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:49:31.977216 13570 hybrid_clock.cc:648] HybridClock initialized: now 1782028171977161 us; error 43 us; skew 500 ppm
Jun 21 07:49:31 dist-test-slave-lzx3 krb5kdc[12536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:49:31.980760 13570 init.cc:377] Logged in from keytab as kudu/127.0.192.193@KRBTEST.COM (short username kudu)
I20260621 07:49:31.981916 13570 webserver.cc:460] Webserver started at http://127.0.192.193:36339/ using document root <none> and password file <none>
I20260621 07:49:31.982587 13570 fs_manager.cc:359] Metadata directory not provided
I20260621 07:49:31.982746 13570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 21 07:49:31 dist-test-slave-lzx3 krb5kdc[12536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for HTTP/127.0.192.212@KRBTEST.COM
I20260621 07:49:32.026809 13570 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.002s sys 0.000s
I20260621 07:49:32.029857 13616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:32.031431 13570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:49:32.031589 13570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-0/wal
uuid: "0e5e14b71ac349ea8747b3dcb01553af"
format_stamp: "Formatted at 2026-06-21 07:49:30 on dist-test-slave-lzx3"
server_key: "96335362c6c9de64bbff0053e8d15ca2"
server_key_iv: "6c93dfadd45fd976dc460196b1738f6d"
server_key_version: "kuduclusterkey@0"
I20260621 07:49:32.032295 13570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:49:32.052644 13570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:49:32.055707 13570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:49:32.055936 13570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:49:32.056624 13570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:49:32.057555 13570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:49:32.057647 13570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:32.057763 13570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:49:32.057842 13570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:32.068060 13570 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.193:38271
I20260621 07:49:32.069357 13570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
I20260621 07:49:32.069514 13729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.193:38271 every 8 connection(s)
I20260621 07:49:32.072762 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 13570
I20260621 07:49:32.073230 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.192.212:57955/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.194:33313
--local_ip_for_outbound_sockets=127.0.192.194
--tserver_master_addrs=127.0.192.254:39807
--webserver_port=42833
--webserver_interface=127.0.192.194
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:43077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.192.212:57955/kms with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 07:49:32 dist-test-slave-lzx3 krb5kdc[12536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:49:32.091935 13733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:32.072416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:39807 (local address 127.0.192.193:55275)
0621 07:49:32.072885 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:32.072928 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 07:49:32.073660 (+ 732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:49:32.075045 (+ 1385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:49:32.075054 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:49:32.075373 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:49:32.076002 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:32.076016 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:32.077493 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:32.077499 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:32.078796 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:32.078809 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:32.079908 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:32.079912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:32.080105 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:32.084727 (+ 4622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:49:32.084745 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:49:32.086340 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:49:32.088737 (+ 2397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:32.088742 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:32.088752 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:32.089102 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:32.089463 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:32.089466 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:32.089467 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:32.089655 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:32.090387 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:49:32.090392 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:49:32.090645 (+ 253us) client_negotiation.cc:769] Sending connection context
0621 07:49:32.090861 (+ 216us) client_negotiation.cc:240] Negotiation successful
0621 07:49:32.091110 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":127,"threads_started":1}
I20260621 07:49:32.093892 13732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:32.072233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:55275 (local address 127.0.192.254:39807)
0621 07:49:32.074026 (+ 1793us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:32.074041 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:32.074069 (+ 28us) server_negotiation.cc:407] Connection header received
0621 07:49:32.074265 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:32.074273 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:32.074619 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:32.076167 (+ 1548us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:49:32.076420 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:32.077374 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:32.078936 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:32.079732 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:32.079914 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:32.086542 (+ 6628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:49:32.086548 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:49:32.086562 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:49:32.086590 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:49:32.088565 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:32.089224 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:32.089229 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:32.089234 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:32.089362 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:32.089784 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:32.089787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:32.089789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:32.090209 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:49:32.090333 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:32.092849 (+ 2516us) server_negotiation.cc:299] Negotiation successful
0621 07:49:32.093104 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1654,"thread_start_us":112,"threads_started":1}
I20260621 07:49:32.096109 13730 heartbeater.cc:347] Connected to a master server at 127.0.192.254:39807
I20260621 07:49:32.096305 13730 heartbeater.cc:464] Registering TS with master...
I20260621 07:49:32.096814 13730 heartbeater.cc:511] Master 127.0.192.254:39807 requested a full tablet report, sending...
W20260621 07:49:32.098803 13730 heartbeater.cc:644] Failed to heartbeat to 127.0.192.254:39807 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260621 07:49:32.098904 13730 heartbeater.cc:464] Registering TS with master...
I20260621 07:49:32.099043 13730 heartbeater.cc:511] Master 127.0.192.254:39807 requested a full tablet report, sending...
I20260621 07:49:32.099939 13730 heartbeater.cc:464] Registering TS with master...
I20260621 07:49:32.100098 13730 heartbeater.cc:511] Master 127.0.192.254:39807 requested a full tablet report, sending...
W20260621 07:49:32.100915 13730 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 07:49:32.220983 13734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:49:32.221215 13734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:49:32.221284 13734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:49:32.225828 13734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:49:32.225935 13734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:49:32.226047 13734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.194
I20260621 07:49:32.232246 13734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:43077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.192.212:57955/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.194:33313
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.192.194
--webserver_port=42833
--tserver_master_addrs=127.0.192.254:39807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.194
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:49:32.233551 13734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:49:32.234472 13734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:49:32.244048 13740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:49:32.244088 13739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:49:32.244369 13742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:49:32.244971 13734 server_base.cc:1034] running on GCE node
I20260621 07:49:32.245303 13734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:49:32.245869 13734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:49:32.247033 13734 hybrid_clock.cc:648] HybridClock initialized: now 1782028172247004 us; error 39 us; skew 500 ppm
Jun 21 07:49:32 dist-test-slave-lzx3 krb5kdc[12536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:49:32.249693 13734 init.cc:377] Logged in from keytab as kudu/127.0.192.194@KRBTEST.COM (short username kudu)
I20260621 07:49:32.250547 13734 webserver.cc:460] Webserver started at http://127.0.192.194:42833/ using document root <none> and password file <none>
I20260621 07:49:32.251143 13734 fs_manager.cc:359] Metadata directory not provided
I20260621 07:49:32.251245 13734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 21 07:49:32 dist-test-slave-lzx3 krb5kdc[12536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for HTTP/127.0.192.212@KRBTEST.COM
I20260621 07:49:32.273851 13734 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260621 07:49:32.277912 13750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:32.279206 13734 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260621 07:49:32.279311 13734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-1/wal
uuid: "ff83fda783ad4c188b9bb8427d96de45"
format_stamp: "Formatted at 2026-06-21 07:49:30 on dist-test-slave-lzx3"
server_key: "bcadaa1e76a6edc6e54020932c60011c"
server_key_iv: "f01785852453e182f4c0404ec32f50fe"
server_key_version: "kuduclusterkey@0"
I20260621 07:49:32.279983 13734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:49:32.324766 13734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:49:32.328289 13734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:49:32.328490 13734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:49:32.329155 13734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:49:32.330196 13734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:49:32.330267 13734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:32.330328 13734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:49:32.330369 13734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:32.342732 13734 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.194:33313
I20260621 07:49:32.343902 13734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
I20260621 07:49:32.344607 13863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.194:33313 every 8 connection(s)
I20260621 07:49:32.352663 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 13734
I20260621 07:49:32.353055 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.192.212:57955/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.195:46451
--local_ip_for_outbound_sockets=127.0.192.195
--tserver_master_addrs=127.0.192.254:39807
--webserver_port=37697
--webserver_interface=127.0.192.195
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:43077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.192.212:57955/kms with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 07:49:32 dist-test-slave-lzx3 krb5kdc[12536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:49:32.363791 13732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:32.345868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:33097 (local address 127.0.192.254:39807)
0621 07:49:32.345981 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:32.345985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:32.347014 (+ 1029us) server_negotiation.cc:407] Connection header received
0621 07:49:32.348295 (+ 1281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:32.348298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:32.348350 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:32.348444 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:49:32.349624 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:32.350442 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:32.351981 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:32.352282 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:32.352361 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:32.360101 (+ 7740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:49:32.360105 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:49:32.360108 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:49:32.360140 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:49:32.361720 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:32.362254 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:32.362258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:32.362260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:32.362372 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:32.362718 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:32.362722 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:32.362724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:32.362895 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:49:32.362998 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:32.363529 (+ 531us) server_negotiation.cc:299] Negotiation successful
0621 07:49:32.363636 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260621 07:49:32.364439 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:32.346338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:39807 (local address 127.0.192.194:33097)
0621 07:49:32.346876 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:32.346925 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0621 07:49:32.348044 (+ 1119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:49:32.348638 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:49:32.348646 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:49:32.348922 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:49:32.349437 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:32.349451 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:32.350603 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:32.350606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:32.351864 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:32.351873 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:32.354718 (+ 2845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:32.354721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:32.354900 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:32.355465 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:49:32.355483 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:49:32.359906 (+ 4423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:49:32.361815 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:32.361820 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:32.361831 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:32.362161 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:32.362497 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:32.362500 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:32.362501 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:32.362627 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:32.363006 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:49:32.363013 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:49:32.363268 (+ 255us) client_negotiation.cc:769] Sending connection context
0621 07:49:32.363490 (+ 222us) client_negotiation.cc:240] Negotiation successful
0621 07:49:32.363748 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":152,"threads_started":1}
I20260621 07:49:32.365486 13864 heartbeater.cc:347] Connected to a master server at 127.0.192.254:39807
I20260621 07:49:32.365681 13864 heartbeater.cc:464] Registering TS with master...
I20260621 07:49:32.366089 13864 heartbeater.cc:511] Master 127.0.192.254:39807 requested a full tablet report, sending...
W20260621 07:49:32.367427 13864 heartbeater.cc:644] Failed to heartbeat to 127.0.192.254:39807 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260621 07:49:32.367544 13864 heartbeater.cc:464] Registering TS with master...
I20260621 07:49:32.367713 13864 heartbeater.cc:511] Master 127.0.192.254:39807 requested a full tablet report, sending...
I20260621 07:49:32.368424 13864 heartbeater.cc:464] Registering TS with master...
I20260621 07:49:32.368595 13864 heartbeater.cc:511] Master 127.0.192.254:39807 requested a full tablet report, sending...
W20260621 07:49:32.369215 13864 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 07:49:32.484843 13868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:49:32.485169 13868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:49:32.485278 13868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:49:32.489859 13868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:49:32.489939 13868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:49:32.490039 13868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.195
I20260621 07:49:32.494697 13868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:43077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.192.212:57955/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.195:46451
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.192.195
--webserver_port=37697
--tserver_master_addrs=127.0.192.254:39807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.195
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:49:32.495654 13868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:49:32.496731 13868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:49:32.506064 13874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:49:32.506181 13876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:49:32.506078 13873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:49:32.506883 13868 server_base.cc:1034] running on GCE node
I20260621 07:49:32.507285 13868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:49:32.507858 13868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:49:32.509063 13868 hybrid_clock.cc:648] HybridClock initialized: now 1782028172509042 us; error 40 us; skew 500 ppm
Jun 21 07:49:32 dist-test-slave-lzx3 krb5kdc[12536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:49:32.511868 13868 init.cc:377] Logged in from keytab as kudu/127.0.192.195@KRBTEST.COM (short username kudu)
I20260621 07:49:32.512828 13868 webserver.cc:460] Webserver started at http://127.0.192.195:37697/ using document root <none> and password file <none>
I20260621 07:49:32.513311 13868 fs_manager.cc:359] Metadata directory not provided
I20260621 07:49:32.513403 13868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 21 07:49:32 dist-test-slave-lzx3 krb5kdc[12536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for HTTP/127.0.192.212@KRBTEST.COM
I20260621 07:49:32.535727 13868 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260621 07:49:32.538945 13884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:32.542243 13868 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260621 07:49:32.542376 13868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-2/wal
uuid: "c459bfb8155f4289a67d84e9f19dce87"
format_stamp: "Formatted at 2026-06-21 07:49:31 on dist-test-slave-lzx3"
server_key: "ed8f817c1b4c26c3762daee15bacd808"
server_key_iv: "2628b956baa871e38a546e11e0869a86"
server_key_version: "kuduclusterkey@0"
I20260621 07:49:32.543053 13868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:49:32.559971 13868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:49:32.563230 13868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:49:32.563419 13868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:49:32.564038 13868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:49:32.565038 13868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:49:32.565093 13868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:32.565145 13868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:49:32.565172 13868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:32.579092 13868 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.195:46451
I20260621 07:49:32.579236 13997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.195:46451 every 8 connection(s)
I20260621 07:49:32.580099 13868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
I20260621 07:49:32.587297 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 13868
Jun 21 07:49:32 dist-test-slave-lzx3 krb5kdc[12536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:49:32.594913 13732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:32.581910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:48385 (local address 127.0.192.254:39807)
0621 07:49:32.582043 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:32.582047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:32.582910 (+ 863us) server_negotiation.cc:407] Connection header received
0621 07:49:32.583939 (+ 1029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:32.583943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:32.584003 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:32.584086 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:49:32.585291 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:32.585820 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:32.586923 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:32.587253 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:32.587370 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:32.590981 (+ 3611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:49:32.590984 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:49:32.590987 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:49:32.591010 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:49:32.592621 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:32.593221 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:32.593227 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:32.593231 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:32.593316 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:32.593737 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:32.593743 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:32.593746 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:32.593933 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:49:32.594046 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:32.594626 (+ 580us) server_negotiation.cc:299] Negotiation successful
0621 07:49:32.594749 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 07:49:32.596124 14000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:32.582327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:39807 (local address 127.0.192.195:48385)
0621 07:49:32.582770 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:32.582818 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0621 07:49:32.583656 (+ 838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:49:32.584264 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:49:32.584271 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:49:32.584613 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:49:32.585158 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:32.585171 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:32.585954 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:32.585957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:32.586801 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:32.586809 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:32.587430 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:32.587435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:32.587649 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:32.588314 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:49:32.588333 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:49:32.590809 (+ 2476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:49:32.592761 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:32.592767 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:32.592780 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:32.593112 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:32.593408 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:32.593411 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:32.593413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:32.593586 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:32.594043 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:49:32.594048 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:49:32.594358 (+ 310us) client_negotiation.cc:769] Sending connection context
0621 07:49:32.594586 (+ 228us) client_negotiation.cc:240] Negotiation successful
0621 07:49:32.594858 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":134,"threads_started":1}
I20260621 07:49:32.596792 14002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:32.587892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60522 (local address 127.0.192.254:39807)
0621 07:49:32.588168 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:32.588176 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:32.588351 (+ 175us) server_negotiation.cc:407] Connection header received
0621 07:49:32.588512 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:32.588515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:32.588595 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:32.588690 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:49:32.589577 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:32.590384 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:32.591333 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:32.591712 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:32.591803 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:32.593406 (+ 1603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:49:32.593408 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:49:32.593410 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:49:32.593423 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:49:32.595063 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:32.595520 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:32.595523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:32.595525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:32.595615 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:32.595914 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:32.595917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:32.595919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:32.596110 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:49:32.596295 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:32.596478 (+ 183us) server_negotiation.cc:299] Negotiation successful
0621 07:49:32.596626 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":110,"threads_started":1}
I20260621 07:49:32.597698 13998 heartbeater.cc:347] Connected to a master server at 127.0.192.254:39807
I20260621 07:49:32.597930 13998 heartbeater.cc:464] Registering TS with master...
I20260621 07:49:32.598475 13998 heartbeater.cc:511] Master 127.0.192.254:39807 requested a full tablet report, sending...
W20260621 07:49:32.600090 13998 heartbeater.cc:644] Failed to heartbeat to 127.0.192.254:39807 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260621 07:49:32.600236 13998 heartbeater.cc:464] Registering TS with master...
I20260621 07:49:32.600412 13998 heartbeater.cc:511] Master 127.0.192.254:39807 requested a full tablet report, sending...
I20260621 07:49:32.601223 13998 heartbeater.cc:464] Registering TS with master...
I20260621 07:49:32.601389 13998 heartbeater.cc:511] Master 127.0.192.254:39807 requested a full tablet report, sending...
W20260621 07:49:32.602078 13998 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 21 07:49:32 dist-test-slave-lzx3 krb5kdc[12536](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782028172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:49:33.101119 13730 heartbeater.cc:464] Registering TS with master...
I20260621 07:49:33.101390 13730 heartbeater.cc:511] Master 127.0.192.254:39807 requested a full tablet report, sending...
I20260621 07:49:33.102442 13532 ts_manager.cc:194] Registered new tserver with Master: 0e5e14b71ac349ea8747b3dcb01553af (127.0.192.193:38271)
I20260621 07:49:33.104020 13532 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.193@KRBTEST.COM'} at 127.0.192.193:55275
Jun 21 07:49:33 dist-test-slave-lzx3 krb5kdc[12536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for HTTP/127.0.192.212@KRBTEST.COM
I20260621 07:49:33.369467 13864 heartbeater.cc:464] Registering TS with master...
I20260621 07:49:33.369756 13864 heartbeater.cc:511] Master 127.0.192.254:39807 requested a full tablet report, sending...
I20260621 07:49:33.370467 13532 ts_manager.cc:194] Registered new tserver with Master: ff83fda783ad4c188b9bb8427d96de45 (127.0.192.194:33313)
I20260621 07:49:33.371061 13532 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.194@KRBTEST.COM'} at 127.0.192.194:33097
Jun 21 07:49:33 dist-test-slave-lzx3 krb5kdc[12536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for HTTP/127.0.192.212@KRBTEST.COM
I20260621 07:49:33.602316 13998 heartbeater.cc:464] Registering TS with master...
I20260621 07:49:33.602609 13998 heartbeater.cc:511] Master 127.0.192.254:39807 requested a full tablet report, sending...
I20260621 07:49:33.603328 13532 ts_manager.cc:194] Registered new tserver with Master: c459bfb8155f4289a67d84e9f19dce87 (127.0.192.195:46451)
I20260621 07:49:33.604146 13532 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.195@KRBTEST.COM'} at 127.0.192.195:48385
I20260621 07:49:33.612111 771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:49:33.623785 14021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:33.613939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60528 (local address 127.0.192.254:39807)
0621 07:49:33.614229 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:33.614234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:33.614618 (+ 384us) server_negotiation.cc:407] Connection header received
0621 07:49:33.614933 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:33.614937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:33.614995 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:33.615102 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:49:33.615936 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:33.616779 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:33.618060 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:33.618420 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:33.618560 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:33.620214 (+ 1654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:49:33.620218 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:49:33.620221 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:49:33.620245 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:49:33.622029 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:33.622489 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:33.622492 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:33.622494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:33.622585 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:33.622875 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:33.622878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:33.622880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:33.623071 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:49:33.623169 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:33.623486 (+ 317us) server_negotiation.cc:299] Negotiation successful
0621 07:49:33.623613 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":113,"threads_started":1}
I20260621 07:49:33.628484 13532 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60528:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:49:33.633318 13532 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:49:33.655078 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:33.647260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:33313 (local address 127.0.0.1:37070)
0621 07:49:33.649233 (+ 1973us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:33.649247 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:49:33.649326 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:49:33.650286 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:49:33.650292 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:49:33.650312 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:49:33.650422 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:33.650428 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:33.651860 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:33.651863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:33.653163 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:33.653174 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:33.654558 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:33.654562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:33.654666 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:33.654702 (+ 36us) client_negotiation.cc:769] Sending connection context
0621 07:49:33.654820 (+ 118us) client_negotiation.cc:240] Negotiation successful
0621 07:49:33.654922 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1833,"thread_start_us":105,"threads_started":1}
I20260621 07:49:33.656925 14030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:33.647454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37070 (local address 127.0.192.194:33313)
0621 07:49:33.649556 (+ 2102us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:33.649563 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:33.649585 (+ 22us) server_negotiation.cc:407] Connection header received
0621 07:49:33.649657 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:33.649663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:33.649857 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:33.650102 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:49:33.650557 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:33.651734 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:33.653463 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:33.654261 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:33.654356 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:33.654485 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:33.656696 (+ 2211us) server_negotiation.cc:299] Negotiation successful
0621 07:49:33.656791 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1988,"thread_start_us":104,"threads_started":1}
I20260621 07:49:33.660046 13780 tablet_service.cc:1467] Processing CreateTablet for tablet f71d793ed4eb49519a480a720d9d0cb8 (DEFAULT_TABLE table=test-table [id=9e77605cdf63485ebc741bc1e6f954a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:49:33.661082 13780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f71d793ed4eb49519a480a720d9d0cb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:49:33.662166 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:33.647531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.195:46451 (local address 127.0.0.1:53232)
0621 07:49:33.655708 (+ 8177us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:33.655721 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:49:33.655783 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:49:33.656411 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:49:33.656414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:49:33.656432 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:49:33.656507 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:33.656513 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:33.658434 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:33.658437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:33.659491 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:33.659501 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:33.661806 (+ 2305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:33.661809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:33.661917 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:33.661938 (+ 21us) client_negotiation.cc:769] Sending connection context
0621 07:49:33.661999 (+ 61us) client_negotiation.cc:240] Negotiation successful
0621 07:49:33.662056 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8129,"thread_start_us":75,"threads_started":1}
I20260621 07:49:33.662277 14032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:33.647884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53232 (local address 127.0.192.195:46451)
0621 07:49:33.653410 (+ 5526us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:33.653416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:33.655868 (+ 2452us) server_negotiation.cc:407] Connection header received
0621 07:49:33.655942 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:33.655947 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:33.656135 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:33.656257 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:49:33.656672 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:33.658298 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:33.659634 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:33.661683 (+ 2049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:33.661795 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:33.661986 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:33.662042 (+ 56us) server_negotiation.cc:299] Negotiation successful
0621 07:49:33.662133 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5434,"thread_start_us":77,"threads_started":1}
I20260621 07:49:33.665108 14027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:33.646139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:38271 (local address 127.0.0.1:51654)
0621 07:49:33.648761 (+ 2622us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:33.648807 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 07:49:33.648944 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:49:33.656159 (+ 7215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:49:33.656162 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:49:33.656180 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:49:33.656280 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:33.656289 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:33.657839 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:33.657843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:33.658974 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:33.658984 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:33.664717 (+ 5733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:33.664722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:33.664816 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:33.664838 (+ 22us) client_negotiation.cc:769] Sending connection context
0621 07:49:33.664908 (+ 70us) client_negotiation.cc:240] Negotiation successful
0621 07:49:33.664974 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2442,"thread_start_us":111,"threads_started":1}
I20260621 07:49:33.665148 14028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:33.646305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51654 (local address 127.0.192.193:38271)
0621 07:49:33.655657 (+ 9352us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:33.655664 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:33.655690 (+ 26us) server_negotiation.cc:407] Connection header received
0621 07:49:33.655770 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:33.655776 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:33.655984 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:33.656442 (+ 458us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:49:33.656619 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:33.657698 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:33.659135 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:33.659951 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:33.664723 (+ 4772us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:33.664862 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:33.664931 (+ 69us) server_negotiation.cc:299] Negotiation successful
0621 07:49:33.665014 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9240,"thread_start_us":140,"threads_started":1}
I20260621 07:49:33.665019 13932 tablet_service.cc:1467] Processing CreateTablet for tablet f71d793ed4eb49519a480a720d9d0cb8 (DEFAULT_TABLE table=test-table [id=9e77605cdf63485ebc741bc1e6f954a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:49:33.665926 13932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f71d793ed4eb49519a480a720d9d0cb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:49:33.667820 13664 tablet_service.cc:1467] Processing CreateTablet for tablet f71d793ed4eb49519a480a720d9d0cb8 (DEFAULT_TABLE table=test-table [id=9e77605cdf63485ebc741bc1e6f954a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:49:33.668757 13664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f71d793ed4eb49519a480a720d9d0cb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:49:33.670048 14033 tablet_bootstrap.cc:492] T f71d793ed4eb49519a480a720d9d0cb8 P ff83fda783ad4c188b9bb8427d96de45: Bootstrap starting.
I20260621 07:49:33.672156 14033 tablet_bootstrap.cc:654] T f71d793ed4eb49519a480a720d9d0cb8 P ff83fda783ad4c188b9bb8427d96de45: Neither blocks nor log segments found. Creating new log.
I20260621 07:49:33.672958 14033 log.cc:824] T f71d793ed4eb49519a480a720d9d0cb8 P ff83fda783ad4c188b9bb8427d96de45: Log is configured to *not* fsync() on all Append() calls
I20260621 07:49:33.673911 14034 tablet_bootstrap.cc:492] T f71d793ed4eb49519a480a720d9d0cb8 P c459bfb8155f4289a67d84e9f19dce87: Bootstrap starting.
I20260621 07:49:33.675519 14034 tablet_bootstrap.cc:654] T f71d793ed4eb49519a480a720d9d0cb8 P c459bfb8155f4289a67d84e9f19dce87: Neither blocks nor log segments found. Creating new log.
I20260621 07:49:33.676122 14034 log.cc:824] T f71d793ed4eb49519a480a720d9d0cb8 P c459bfb8155f4289a67d84e9f19dce87: Log is configured to *not* fsync() on all Append() calls
I20260621 07:49:33.678691 14033 tablet_bootstrap.cc:492] T f71d793ed4eb49519a480a720d9d0cb8 P ff83fda783ad4c188b9bb8427d96de45: No bootstrap required, opened a new log
I20260621 07:49:33.678952 14033 ts_tablet_manager.cc:1397] T f71d793ed4eb49519a480a720d9d0cb8 P ff83fda783ad4c188b9bb8427d96de45: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260621 07:49:33.682979 14033 raft_consensus.cc:348] T f71d793ed4eb49519a480a720d9d0cb8 P ff83fda783ad4c188b9bb8427d96de45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5e14b71ac349ea8747b3dcb01553af" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 38271 } } peers { permanent_uuid: "ff83fda783ad4c188b9bb8427d96de45" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 33313 } } peers { permanent_uuid: "c459bfb8155f4289a67d84e9f19dce87" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 46451 } }
I20260621 07:49:33.683200 14033 raft_consensus.cc:374] T f71d793ed4eb49519a480a720d9d0cb8 P ff83fda783ad4c188b9bb8427d96de45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:49:33.683255 14033 raft_consensus.cc:729] T f71d793ed4eb49519a480a720d9d0cb8 P ff83fda783ad4c188b9bb8427d96de45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff83fda783ad4c188b9bb8427d96de45, State: Initialized, Role: FOLLOWER
I20260621 07:49:33.683658 14033 consensus_queue.cc:260] T f71d793ed4eb49519a480a720d9d0cb8 P ff83fda783ad4c188b9bb8427d96de45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5e14b71ac349ea8747b3dcb01553af" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 38271 } } peers { permanent_uuid: "ff83fda783ad4c188b9bb8427d96de45" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 33313 } } peers { permanent_uuid: "c459bfb8155f4289a67d84e9f19dce87" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 46451 } }
I20260621 07:49:33.684209 13864 heartbeater.cc:503] Master 127.0.192.254:39807 was elected leader, sending a full tablet report...
I20260621 07:49:33.685940 14034 tablet_bootstrap.cc:492] T f71d793ed4eb49519a480a720d9d0cb8 P c459bfb8155f4289a67d84e9f19dce87: No bootstrap required, opened a new log
I20260621 07:49:33.686117 14034 ts_tablet_manager.cc:1397] T f71d793ed4eb49519a480a720d9d0cb8 P c459bfb8155f4289a67d84e9f19dce87: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20260621 07:49:33.689882 14034 raft_consensus.cc:348] T f71d793ed4eb49519a480a720d9d0cb8 P c459bfb8155f4289a67d84e9f19dce87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5e14b71ac349ea8747b3dcb01553af" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 38271 } } peers { permanent_uuid: "ff83fda783ad4c188b9bb8427d96de45" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 33313 } } peers { permanent_uuid: "c459bfb8155f4289a67d84e9f19dce87" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 46451 } }
I20260621 07:49:33.690119 14034 raft_consensus.cc:374] T f71d793ed4eb49519a480a720d9d0cb8 P c459bfb8155f4289a67d84e9f19dce87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:49:33.690187 14034 raft_consensus.cc:729] T f71d793ed4eb49519a480a720d9d0cb8 P c459bfb8155f4289a67d84e9f19dce87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c459bfb8155f4289a67d84e9f19dce87, State: Initialized, Role: FOLLOWER
I20260621 07:49:33.690604 14034 consensus_queue.cc:260] T f71d793ed4eb49519a480a720d9d0cb8 P c459bfb8155f4289a67d84e9f19dce87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5e14b71ac349ea8747b3dcb01553af" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 38271 } } peers { permanent_uuid: "ff83fda783ad4c188b9bb8427d96de45" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 33313 } } peers { permanent_uuid: "c459bfb8155f4289a67d84e9f19dce87" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 46451 } }
I20260621 07:49:33.690852 14038 tablet_bootstrap.cc:492] T f71d793ed4eb49519a480a720d9d0cb8 P 0e5e14b71ac349ea8747b3dcb01553af: Bootstrap starting.
I20260621 07:49:33.691283 13998 heartbeater.cc:503] Master 127.0.192.254:39807 was elected leader, sending a full tablet report...
I20260621 07:49:33.691553 14034 ts_tablet_manager.cc:1428] T f71d793ed4eb49519a480a720d9d0cb8 P c459bfb8155f4289a67d84e9f19dce87: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260621 07:49:33.691879 14033 ts_tablet_manager.cc:1428] T f71d793ed4eb49519a480a720d9d0cb8 P ff83fda783ad4c188b9bb8427d96de45: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20260621 07:49:33.695858 14038 tablet_bootstrap.cc:654] T f71d793ed4eb49519a480a720d9d0cb8 P 0e5e14b71ac349ea8747b3dcb01553af: Neither blocks nor log segments found. Creating new log.
I20260621 07:49:33.696743 14038 log.cc:824] T f71d793ed4eb49519a480a720d9d0cb8 P 0e5e14b71ac349ea8747b3dcb01553af: Log is configured to *not* fsync() on all Append() calls
I20260621 07:49:33.698688 14038 tablet_bootstrap.cc:492] T f71d793ed4eb49519a480a720d9d0cb8 P 0e5e14b71ac349ea8747b3dcb01553af: No bootstrap required, opened a new log
I20260621 07:49:33.698926 14038 ts_tablet_manager.cc:1397] T f71d793ed4eb49519a480a720d9d0cb8 P 0e5e14b71ac349ea8747b3dcb01553af: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260621 07:49:33.702930 14038 raft_consensus.cc:348] T f71d793ed4eb49519a480a720d9d0cb8 P 0e5e14b71ac349ea8747b3dcb01553af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5e14b71ac349ea8747b3dcb01553af" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 38271 } } peers { permanent_uuid: "ff83fda783ad4c188b9bb8427d96de45" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 33313 } } peers { permanent_uuid: "c459bfb8155f4289a67d84e9f19dce87" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 46451 } }
I20260621 07:49:33.703236 14038 raft_consensus.cc:374] T f71d793ed4eb49519a480a720d9d0cb8 P 0e5e14b71ac349ea8747b3dcb01553af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:49:33.703332 14038 raft_consensus.cc:729] T f71d793ed4eb49519a480a720d9d0cb8 P 0e5e14b71ac349ea8747b3dcb01553af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e5e14b71ac349ea8747b3dcb01553af, State: Initialized, Role: FOLLOWER
I20260621 07:49:33.703826 14038 consensus_queue.cc:260] T f71d793ed4eb49519a480a720d9d0cb8 P 0e5e14b71ac349ea8747b3dcb01553af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5e14b71ac349ea8747b3dcb01553af" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 38271 } } peers { permanent_uuid: "ff83fda783ad4c188b9bb8427d96de45" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 33313 } } peers { permanent_uuid: "c459bfb8155f4289a67d84e9f19dce87" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 46451 } }
I20260621 07:49:33.704813 14038 ts_tablet_manager.cc:1428] T f71d793ed4eb49519a480a720d9d0cb8 P 0e5e14b71ac349ea8747b3dcb01553af: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260621 07:49:33.705143 13730 heartbeater.cc:503] Master 127.0.192.254:39807 was elected leader, sending a full tablet report...
I20260621 07:49:33.724050 14041 raft_consensus.cc:482] T f71d793ed4eb49519a480a720d9d0cb8 P 0e5e14b71ac349ea8747b3dcb01553af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:49:33.724284 14041 raft_consensus.cc:504] T f71d793ed4eb49519a480a720d9d0cb8 P 0e5e14b71ac349ea8747b3dcb01553af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5e14b71ac349ea8747b3dcb01553af" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 38271 } } peers { permanent_uuid: "ff83fda783ad4c188b9bb8427d96de45" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 33313 } } peers { permanent_uuid: "c459bfb8155f4289a67d84e9f19dce87" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 46451 } }
I20260621 07:49:33.726537 14041 leader_election.cc:288] T f71d793ed4eb49519a480a720d9d0cb8 P 0e5e14b71ac349ea8747b3dcb01553af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff83fda783ad4c188b9bb8427d96de45 (127.0.192.194:33313), c459bfb8155f4289a67d84e9f19dce87 (127.0.192.195:46451)
I20260621 07:49:33.731737 14030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:33.726276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:55321 (local address 127.0.192.194:33313)
0621 07:49:33.726391 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:33.726395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:33.727203 (+ 808us) server_negotiation.cc:407] Connection header received
0621 07:49:33.727549 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:33.727552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:33.727600 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:33.727697 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:49:33.728126 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:33.728930 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:33.730532 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:33.731089 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:33.731176 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:33.731224 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:33.731547 (+ 323us) server_negotiation.cc:299] Negotiation successful
0621 07:49:33.731617 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260621 07:49:33.732003 14032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:33.726019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:52971 (local address 127.0.192.195:46451)
0621 07:49:33.726177 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:33.726183 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:33.727759 (+ 1576us) server_negotiation.cc:407] Connection header received
0621 07:49:33.727946 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:33.727949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:33.727997 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:33.728046 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:49:33.728488 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:33.729290 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:33.730789 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:33.731340 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:33.731445 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:33.731492 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:33.731809 (+ 317us) server_negotiation.cc:299] Negotiation successful
0621 07:49:33.731884 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260621 07:49:33.732435 14042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:33.725830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.195:46451 (local address 127.0.192.193:52971)
0621 07:49:33.727672 (+ 1842us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:33.727687 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:49:33.727823 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:49:33.728182 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:49:33.728185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:49:33.728201 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:49:33.728338 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:33.728345 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:33.729461 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:33.729465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:33.730633 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:33.730644 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:33.731582 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:33.731586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:33.731677 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:33.731698 (+ 21us) client_negotiation.cc:769] Sending connection context
0621 07:49:33.732183 (+ 485us) client_negotiation.cc:240] Negotiation successful
0621 07:49:33.732255 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1763,"mutex_wait_us":248,"thread_start_us":96,"threads_started":1}
I20260621 07:49:33.732754 14043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:33.725826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:33313 (local address 127.0.192.193:55321)
0621 07:49:33.726976 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:33.726992 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:49:33.727408 (+ 416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:49:33.727821 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:49:33.727824 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:49:33.727844 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:49:33.727993 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:33.727999 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:33.729092 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:33.729095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:33.730401 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:33.730412 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:33.731310 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:33.731313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:33.731409 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:33.731431 (+ 22us) client_negotiation.cc:769] Sending connection context
0621 07:49:33.732397 (+ 966us) client_negotiation.cc:240] Negotiation successful
0621 07:49:33.732472 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1074,"thread_start_us":122,"threads_started":1}
I20260621 07:49:33.733069 13952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f71d793ed4eb49519a480a720d9d0cb8" candidate_uuid: "0e5e14b71ac349ea8747b3dcb01553af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c459bfb8155f4289a67d84e9f19dce87" is_pre_election: true
I20260621 07:49:33.733408 13952 raft_consensus.cc:2454] T f71d793ed4eb49519a480a720d9d0cb8 P c459bfb8155f4289a67d84e9f19dce87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e5e14b71ac349ea8747b3dcb01553af in term 0.
I20260621 07:49:33.733429 13813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f71d793ed4eb49519a480a720d9d0cb8" candidate_uuid: "0e5e14b71ac349ea8747b3dcb01553af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff83fda783ad4c188b9bb8427d96de45" is_pre_election: true
I20260621 07:49:33.733682 13813 raft_consensus.cc:2454] T f71d793ed4eb49519a480a720d9d0cb8 P ff83fda783ad4c188b9bb8427d96de45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e5e14b71ac349ea8747b3dcb01553af in term 0.
I20260621 07:49:33.733913 13617 leader_election.cc:302] T f71d793ed4eb49519a480a720d9d0cb8 P 0e5e14b71ac349ea8747b3dcb01553af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e5e14b71ac349ea8747b3dcb01553af, c459bfb8155f4289a67d84e9f19dce87; no voters:
I20260621 07:49:33.734212 14041 raft_consensus.cc:2781] T f71d793ed4eb49519a480a720d9d0cb8 P 0e5e14b71ac349ea8747b3dcb01553af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:49:33.734303 14041 raft_consensus.cc:482] T f71d793ed4eb49519a480a720d9d0cb8 P 0e5e14b71ac349ea8747b3dcb01553af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:49:33.734372 14041 raft_consensus.cc:3037] T f71d793ed4eb49519a480a720d9d0cb8 P 0e5e14b71ac349ea8747b3dcb01553af [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:49:33.735702 14041 raft_consensus.cc:504] T f71d793ed4eb49519a480a720d9d0cb8 P 0e5e14b71ac349ea8747b3dcb01553af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5e14b71ac349ea8747b3dcb01553af" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 38271 } } peers { permanent_uuid: "ff83fda783ad4c188b9bb8427d96de45" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 33313 } } peers { permanent_uuid: "c459bfb8155f4289a67d84e9f19dce87" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 46451 } }
I20260621 07:49:33.736053 14041 leader_election.cc:288] T f71d793ed4eb49519a480a720d9d0cb8 P 0e5e14b71ac349ea8747b3dcb01553af [CANDIDATE]: Term 1 election: Requested vote from peers ff83fda783ad4c188b9bb8427d96de45 (127.0.192.194:33313), c459bfb8155f4289a67d84e9f19dce87 (127.0.192.195:46451)
I20260621 07:49:33.736413 13813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f71d793ed4eb49519a480a720d9d0cb8" candidate_uuid: "0e5e14b71ac349ea8747b3dcb01553af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff83fda783ad4c188b9bb8427d96de45"
I20260621 07:49:33.736475 13952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f71d793ed4eb49519a480a720d9d0cb8" candidate_uuid: "0e5e14b71ac349ea8747b3dcb01553af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c459bfb8155f4289a67d84e9f19dce87"
I20260621 07:49:33.736519 13813 raft_consensus.cc:3037] T f71d793ed4eb49519a480a720d9d0cb8 P ff83fda783ad4c188b9bb8427d96de45 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:49:33.736618 13952 raft_consensus.cc:3037] T f71d793ed4eb49519a480a720d9d0cb8 P c459bfb8155f4289a67d84e9f19dce87 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:49:33.737776 13813 raft_consensus.cc:2454] T f71d793ed4eb49519a480a720d9d0cb8 P ff83fda783ad4c188b9bb8427d96de45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e5e14b71ac349ea8747b3dcb01553af in term 1.
I20260621 07:49:33.737918 13952 raft_consensus.cc:2454] T f71d793ed4eb49519a480a720d9d0cb8 P c459bfb8155f4289a67d84e9f19dce87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e5e14b71ac349ea8747b3dcb01553af in term 1.
I20260621 07:49:33.738159 13619 leader_election.cc:302] T f71d793ed4eb49519a480a720d9d0cb8 P 0e5e14b71ac349ea8747b3dcb01553af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e5e14b71ac349ea8747b3dcb01553af, ff83fda783ad4c188b9bb8427d96de45; no voters:
I20260621 07:49:33.738440 14041 raft_consensus.cc:2781] T f71d793ed4eb49519a480a720d9d0cb8 P 0e5e14b71ac349ea8747b3dcb01553af [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:49:33.738660 14041 raft_consensus.cc:686] T f71d793ed4eb49519a480a720d9d0cb8 P 0e5e14b71ac349ea8747b3dcb01553af [term 1 LEADER]: Becoming Leader. State: Replica: 0e5e14b71ac349ea8747b3dcb01553af, State: Running, Role: LEADER
I20260621 07:49:33.739019 14041 consensus_queue.cc:237] T f71d793ed4eb49519a480a720d9d0cb8 P 0e5e14b71ac349ea8747b3dcb01553af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5e14b71ac349ea8747b3dcb01553af" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 38271 } } peers { permanent_uuid: "ff83fda783ad4c188b9bb8427d96de45" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 33313 } } peers { permanent_uuid: "c459bfb8155f4289a67d84e9f19dce87" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 46451 } }
I20260621 07:49:33.742700 13531 catalog_manager.cc:5495] T f71d793ed4eb49519a480a720d9d0cb8 P 0e5e14b71ac349ea8747b3dcb01553af reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e5e14b71ac349ea8747b3dcb01553af (127.0.192.193). New cstate: current_term: 1 leader_uuid: "0e5e14b71ac349ea8747b3dcb01553af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5e14b71ac349ea8747b3dcb01553af" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 38271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff83fda783ad4c188b9bb8427d96de45" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 33313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c459bfb8155f4289a67d84e9f19dce87" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 46451 } health_report { overall_health: UNKNOWN } } }
I20260621 07:49:33.772897 14028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:33.768045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51668 (local address 127.0.192.193:38271)
0621 07:49:33.768192 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:33.768198 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:33.768223 (+ 25us) server_negotiation.cc:407] Connection header received
0621 07:49:33.768299 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:33.768301 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:33.768361 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:33.768458 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:49:33.768803 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:33.769634 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:33.770833 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:33.771069 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:33.771187 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:33.771505 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:49:33.772087 (+ 582us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:49:33.772229 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:33.772655 (+ 426us) server_negotiation.cc:299] Negotiation successful
0621 07:49:33.772718 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260621 07:49:33.779716 13952 raft_consensus.cc:1261] T f71d793ed4eb49519a480a720d9d0cb8 P c459bfb8155f4289a67d84e9f19dce87 [term 1 FOLLOWER]: Refusing update from remote peer 0e5e14b71ac349ea8747b3dcb01553af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:49:33.779722 13813 raft_consensus.cc:1261] T f71d793ed4eb49519a480a720d9d0cb8 P ff83fda783ad4c188b9bb8427d96de45 [term 1 FOLLOWER]: Refusing update from remote peer 0e5e14b71ac349ea8747b3dcb01553af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:49:33.780491 14041 consensus_queue.cc:1035] T f71d793ed4eb49519a480a720d9d0cb8 P 0e5e14b71ac349ea8747b3dcb01553af [LEADER]: Connected to new peer: Peer: permanent_uuid: "c459bfb8155f4289a67d84e9f19dce87" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 46451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:49:33.780719 14044 consensus_queue.cc:1035] T f71d793ed4eb49519a480a720d9d0cb8 P 0e5e14b71ac349ea8747b3dcb01553af [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff83fda783ad4c188b9bb8427d96de45" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:49:33.789417 14054 mvcc.cc:205] Tried to move back new op lower bound from 7299187399794995200 to 7299187399638495232. Current Snapshot: MvccSnapshot[applied={T|T < 7299187399794995200}]
I20260621 07:49:33.793998 14056 mvcc.cc:205] Tried to move back new op lower bound from 7299187399794995200 to 7299187399638495232. Current Snapshot: MvccSnapshot[applied={T|T < 7299187399794995200}]
I20260621 07:49:33.799422 14055 mvcc.cc:205] Tried to move back new op lower bound from 7299187399794995200 to 7299187399638495232. Current Snapshot: MvccSnapshot[applied={T|T < 7299187399794995200 or (T in {7299187399794995200})}]
I20260621 07:49:33.800401 13532 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:49:33.800558 13532 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:49:33.803383 13532 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3990f8b0921a42c3ac2abaf0b7287be9: Sending DeleteTablet for 3 replicas of tablet f71d793ed4eb49519a480a720d9d0cb8
I20260621 07:49:33.804319 13664 tablet_service.cc:1514] Processing DeleteTablet for tablet f71d793ed4eb49519a480a720d9d0cb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:49:33 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:51654
I20260621 07:49:33.805119 14063 tablet_replica.cc:330] T f71d793ed4eb49519a480a720d9d0cb8 P 0e5e14b71ac349ea8747b3dcb01553af: stopping tablet replica
I20260621 07:49:33.805243 13780 tablet_service.cc:1514] Processing DeleteTablet for tablet f71d793ed4eb49519a480a720d9d0cb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:49:33 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:37070
I20260621 07:49:33.805567 14063 raft_consensus.cc:2229] T f71d793ed4eb49519a480a720d9d0cb8 P 0e5e14b71ac349ea8747b3dcb01553af [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:49:33.805908 14063 raft_consensus.cc:2258] T f71d793ed4eb49519a480a720d9d0cb8 P 0e5e14b71ac349ea8747b3dcb01553af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:49:33.806080 13932 tablet_service.cc:1514] Processing DeleteTablet for tablet f71d793ed4eb49519a480a720d9d0cb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:49:33 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:53232
I20260621 07:49:33.806433 14064 tablet_replica.cc:330] T f71d793ed4eb49519a480a720d9d0cb8 P ff83fda783ad4c188b9bb8427d96de45: stopping tablet replica
I20260621 07:49:33.806723 14064 raft_consensus.cc:2229] T f71d793ed4eb49519a480a720d9d0cb8 P ff83fda783ad4c188b9bb8427d96de45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:49:33.806880 14065 tablet_replica.cc:330] T f71d793ed4eb49519a480a720d9d0cb8 P c459bfb8155f4289a67d84e9f19dce87: stopping tablet replica
I20260621 07:49:33.807075 14064 raft_consensus.cc:2258] T f71d793ed4eb49519a480a720d9d0cb8 P ff83fda783ad4c188b9bb8427d96de45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:49:33.807301 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 13570
I20260621 07:49:33.807422 14063 ts_tablet_manager.cc:1905] T f71d793ed4eb49519a480a720d9d0cb8 P 0e5e14b71ac349ea8747b3dcb01553af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:49:33.808919 14064 ts_tablet_manager.cc:1905] T f71d793ed4eb49519a480a720d9d0cb8 P ff83fda783ad4c188b9bb8427d96de45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:49:33.809442 14065 raft_consensus.cc:2229] T f71d793ed4eb49519a480a720d9d0cb8 P c459bfb8155f4289a67d84e9f19dce87 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:49:33.809718 14065 raft_consensus.cc:2258] T f71d793ed4eb49519a480a720d9d0cb8 P c459bfb8155f4289a67d84e9f19dce87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:49:33.811031 14065 ts_tablet_manager.cc:1905] T f71d793ed4eb49519a480a720d9d0cb8 P c459bfb8155f4289a67d84e9f19dce87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:49:33.811717 14064 ts_tablet_manager.cc:1918] T f71d793ed4eb49519a480a720d9d0cb8 P ff83fda783ad4c188b9bb8427d96de45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:49:33.811797 14064 log.cc:1196] T f71d793ed4eb49519a480a720d9d0cb8 P ff83fda783ad4c188b9bb8427d96de45: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-1/wal/wals/f71d793ed4eb49519a480a720d9d0cb8
I20260621 07:49:33.812126 14064 ts_tablet_manager.cc:1939] T f71d793ed4eb49519a480a720d9d0cb8 P ff83fda783ad4c188b9bb8427d96de45: Deleting consensus metadata
I20260621 07:49:33.813241 13518 catalog_manager.cc:4841] TS ff83fda783ad4c188b9bb8427d96de45 (127.0.192.194:33313): tablet f71d793ed4eb49519a480a720d9d0cb8 (table test-table [id=9e77605cdf63485ebc741bc1e6f954a9]) successfully deleted
W20260621 07:49:33.814106 13516 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.192.193:38271 (error 108)
I20260621 07:49:33.814586 14065 ts_tablet_manager.cc:1918] T f71d793ed4eb49519a480a720d9d0cb8 P c459bfb8155f4289a67d84e9f19dce87: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:49:33.814673 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 13734
I20260621 07:49:33.814689 14065 log.cc:1196] T f71d793ed4eb49519a480a720d9d0cb8 P c459bfb8155f4289a67d84e9f19dce87: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782028028522580-771-0/minicluster-data/ts-2/wal/wals/f71d793ed4eb49519a480a720d9d0cb8
I20260621 07:49:33.815955 14065 ts_tablet_manager.cc:1939] T f71d793ed4eb49519a480a720d9d0cb8 P c459bfb8155f4289a67d84e9f19dce87: Deleting consensus metadata
I20260621 07:49:33.815992 14027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:33.815690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:38271 (local address 127.0.0.1:51670)
0621 07:49:33.815837 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:33.815909 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.193:38271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260621 07:49:33.816241 13516 catalog_manager.cc:4584] TS 0e5e14b71ac349ea8747b3dcb01553af (127.0.192.193:38271): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f71d793ed4eb49519a480a720d9d0cb8: Network error: Client connection negotiation failed: client connection to 127.0.192.193:38271: connect: Connection refused (error 111)
I20260621 07:49:33.817016 13516 catalog_manager.cc:4841] TS c459bfb8155f4289a67d84e9f19dce87 (127.0.192.195:46451): tablet f71d793ed4eb49519a480a720d9d0cb8 (table test-table [id=9e77605cdf63485ebc741bc1e6f954a9]) successfully deleted
I20260621 07:49:33.820923 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 13868
I20260621 07:49:33.825847 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 13499
I20260621 07:49:33.831645 771 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260621 07:49:34.165692 771 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260621 07:49:34.165784 771 mini_ranger.cc:66] Stopping Ranger...
I20260621 07:49:34.509855 771 mini_ranger.cc:68] Stopped Ranger
2026-06-21 07:49:34.510 UTC [12566] LOG: received smart shutdown request
2026-06-21 07:49:34.513 UTC [12566] LOG: background worker "logical replication launcher" (PID 12575) exited with exit code 1
2026-06-21 07:49:34.513 UTC [12570] LOG: shutting down
2026-06-21 07:49:34.657 UTC [12566] LOG: database system is shut down
2026-06-21T07:49:34Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (43394 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:49:34 dist-test-slave-lzx3 krb5kdc[14075](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:49:34 dist-test-slave-lzx3 krb5kdc[14075](info): set up 2 sockets
Jun 21 07:49:34 dist-test-slave-lzx3 krb5kdc[14075](info): commencing operation
krb5kdc: starting...
W20260621 07:49:35.732273 771 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/krb5kdc/test-user.keytab.
Jun 21 07:49:35 dist-test-slave-lzx3 krb5kdc[14075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:49:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:49:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.192.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
I20260621 07:49:35.842718 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:34803
--webserver_interface=127.0.192.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:38749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:34803,127.0.192.253:42889,127.0.192.252:42621
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:49:35.938366 14088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:49:35.938612 14088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:49:35.938681 14088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:49:35.942507 14088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:49:35.942588 14088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:49:35.942625 14088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:49:35.942682 14088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:49:35.942718 14088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:49:35.947445 14088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:38749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:34803,127.0.192.253:42889,127.0.192.252:42621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:34803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:49:35.948267 14088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:49:35.949141 14088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:49:35.956292 14094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:49:35.956311 14093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:49:35.956437 14096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:49:35.956508 14088 server_base.cc:1034] running on GCE node
I20260621 07:49:35.957067 14088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:49:35.957994 14088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:49:35.959177 14088 hybrid_clock.cc:648] HybridClock initialized: now 1782028175959154 us; error 37 us; skew 500 ppm
Jun 21 07:49:35 dist-test-slave-lzx3 krb5kdc[14075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:49:35.964547 14088 init.cc:377] Logged in from keytab as kudu/127.0.192.254@KRBTEST.COM (short username kudu)
I20260621 07:49:35.965381 14088 webserver.cc:460] Webserver started at http://127.0.192.254:44289/ using document root <none> and password file <none>
I20260621 07:49:35.965863 14088 fs_manager.cc:359] Metadata directory not provided
I20260621 07:49:35.965921 14088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:49:35.966135 14088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:49:35.967860 14088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "359c9c7393414440b566e6e1dcaa6f2c"
format_stamp: "Formatted at 2026-06-21 07:49:35 on dist-test-slave-lzx3"
server_key: "ae2d59e4e0a85b822ffd7e928c7e98ef"
server_key_iv: "19bf1ca92ef4c21463d822586e5b6ef5"
server_key_version: "encryptionkey@0"
I20260621 07:49:35.968410 14088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "359c9c7393414440b566e6e1dcaa6f2c"
format_stamp: "Formatted at 2026-06-21 07:49:35 on dist-test-slave-lzx3"
server_key: "ae2d59e4e0a85b822ffd7e928c7e98ef"
server_key_iv: "19bf1ca92ef4c21463d822586e5b6ef5"
server_key_version: "encryptionkey@0"
I20260621 07:49:35.971467 14088 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260621 07:49:35.973482 14103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:35.974468 14088 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:49:35.974587 14088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "359c9c7393414440b566e6e1dcaa6f2c"
format_stamp: "Formatted at 2026-06-21 07:49:35 on dist-test-slave-lzx3"
server_key: "ae2d59e4e0a85b822ffd7e928c7e98ef"
server_key_iv: "19bf1ca92ef4c21463d822586e5b6ef5"
server_key_version: "encryptionkey@0"
I20260621 07:49:35.974676 14088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:49:35.999480 14088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:49:36.002224 14088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:49:36.002409 14088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:49:36.008829 14088 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:34803
I20260621 07:49:36.008854 14155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:34803 every 8 connection(s)
I20260621 07:49:36.009655 14088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:49:36.012398 14156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:49:36.015442 14156 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34803 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:49:36.017246 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 14088
I20260621 07:49:36.017351 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:49:36.017586 771 external_mini_cluster.cc:1428] Setting key 840773ceca8271a805d754b8a654b2c5
Jun 21 07:49:36 dist-test-slave-lzx3 krb5kdc[14075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.192.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.253@KRBTEST.COM" created.
I20260621 07:49:36.029763 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:36.017232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:34803 (local address 127.0.0.1:45856)
0621 07:49:36.017688 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:36.017763 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0621 07:49:36.018946 (+ 1183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:49:36.019722 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:49:36.019726 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:49:36.020019 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:49:36.020223 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:36.020235 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.021346 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.021352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:36.022548 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:36.022560 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.023138 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.023143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:36.023272 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:36.023739 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:49:36.023760 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:49:36.025322 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:49:36.026970 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:36.026979 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:36.026987 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:36.027184 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:36.027534 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:36.027538 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:36.027541 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:36.027662 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:36.028250 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:49:36.028256 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:49:36.028398 (+ 142us) client_negotiation.cc:769] Sending connection context
0621 07:49:36.028552 (+ 154us) client_negotiation.cc:240] Negotiation successful
0621 07:49:36.028948 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":131,"thread_start_us":125,"threads_started":1}
I20260621 07:49:36.029780 14159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:36.017965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45856 (local address 127.0.192.254:34803)
0621 07:49:36.018223 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:36.018234 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:36.018274 (+ 40us) server_negotiation.cc:407] Connection header received
0621 07:49:36.019296 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:36.019303 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:36.019525 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:36.019715 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:49:36.020367 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.021216 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.022662 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.022997 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.023103 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:36.025461 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:49:36.025466 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:49:36.025474 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:49:36.025492 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:49:36.026870 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:36.027282 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:36.027289 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:36.027290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:36.027419 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:36.027757 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:36.027763 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:36.027767 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:36.028138 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:49:36.028273 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:36.028659 (+ 386us) server_negotiation.cc:299] Negotiation successful
0621 07:49:36.028901 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":152,"thread_start_us":91,"threads_started":1}
I20260621 07:49:36.031898 14156 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.192.253" port: 42889 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:49:36.032516 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:36.032294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.253:42889 (local address 127.0.0.1:54104)
0621 07:49:36.032396 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:36.032465 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.253:42889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260621 07:49:36.032718 14104 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.192.253:42889: connect: Connection refused (error 111)
I20260621 07:49:36.033430 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:36.033259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.253:42889 (local address 127.0.0.1:54106)
0621 07:49:36.033348 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:36.033386 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.253:42889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260621 07:49:36.033620 14156 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.192.253:42889: Network error: Client connection negotiation failed: client connection to 127.0.192.253:42889: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/krb5kdc/kudu_127.0.192.253.keytab.
Entry for principal kudu/127.0.192.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/krb5kdc/kudu_127.0.192.253.keytab.
I20260621 07:49:36.041676 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.253:42889
--webserver_interface=127.0.192.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/krb5kdc/kudu_127.0.192.253.keytab
--principal=kudu/127.0.192.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:38749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:34803,127.0.192.253:42889,127.0.192.252:42621
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:49:36.082818 14156 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.192.253" port: 42889 } attempt: 1
I20260621 07:49:36.083496 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:36.083262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.253:42889 (local address 127.0.0.1:54120)
0621 07:49:36.083384 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:36.083434 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.253:42889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260621 07:49:36.084149 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:36.083965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.253:42889 (local address 127.0.0.1:54130)
0621 07:49:36.084058 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:36.084099 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.253:42889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260621 07:49:36.084326 14156 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.192.253:42889: Network error: Client connection negotiation failed: client connection to 127.0.192.253:42889: connect: Connection refused (error 111)
W20260621 07:49:36.138105 14163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:49:36.138298 14163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:49:36.138334 14163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:49:36.141785 14163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:49:36.141825 14163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:49:36.141842 14163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:49:36.141856 14163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:49:36.141870 14163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:49:36.146461 14163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:38749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:34803,127.0.192.253:42889,127.0.192.252:42621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/krb5kdc/kudu_127.0.192.253.keytab
--principal=kudu/127.0.192.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.253:42889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.192.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:49:36.147282 14163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:49:36.148067 14163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 07:49:36.152493 14156 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.192.253" port: 42889 } attempt: 2
I20260621 07:49:36.153116 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:36.152875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.253:42889 (local address 127.0.0.1:54136)
0621 07:49:36.152987 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:36.153037 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.253:42889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260621 07:49:36.153733 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:36.153545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.253:42889 (local address 127.0.0.1:54140)
0621 07:49:36.153637 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:36.153676 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.253:42889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260621 07:49:36.153874 14156 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.192.253:42889: Network error: Client connection negotiation failed: client connection to 127.0.192.253:42889: connect: Connection refused (error 111)
W20260621 07:49:36.155459 14171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:49:36.155464 14169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:49:36.155490 14168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:49:36.155619 14163 server_base.cc:1034] running on GCE node
I20260621 07:49:36.156244 14163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:49:36.157112 14163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:49:36.158254 14163 hybrid_clock.cc:648] HybridClock initialized: now 1782028176158240 us; error 25 us; skew 500 ppm
Jun 21 07:49:36 dist-test-slave-lzx3 krb5kdc[14075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:49:36.160663 14163 init.cc:377] Logged in from keytab as kudu/127.0.192.253@KRBTEST.COM (short username kudu)
I20260621 07:49:36.161407 14163 webserver.cc:460] Webserver started at http://127.0.192.253:37937/ using document root <none> and password file <none>
I20260621 07:49:36.161872 14163 fs_manager.cc:359] Metadata directory not provided
I20260621 07:49:36.161935 14163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:49:36.162114 14163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:49:36.163659 14163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-1/data/instance:
uuid: "e21c6dc532ca420b9d9a333c9bc6f5c8"
format_stamp: "Formatted at 2026-06-21 07:49:36 on dist-test-slave-lzx3"
server_key: "90221420ad66581f642a0385afa40932"
server_key_iv: "1fe64176a9058fa37f07303f4879bfa2"
server_key_version: "encryptionkey@0"
I20260621 07:49:36.164096 14163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-1/wal/instance:
uuid: "e21c6dc532ca420b9d9a333c9bc6f5c8"
format_stamp: "Formatted at 2026-06-21 07:49:36 on dist-test-slave-lzx3"
server_key: "90221420ad66581f642a0385afa40932"
server_key_iv: "1fe64176a9058fa37f07303f4879bfa2"
server_key_version: "encryptionkey@0"
I20260621 07:49:36.167218 14163 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 07:49:36.169226 14178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:36.170222 14163 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:49:36.170331 14163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-1/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-1/wal
uuid: "e21c6dc532ca420b9d9a333c9bc6f5c8"
format_stamp: "Formatted at 2026-06-21 07:49:36 on dist-test-slave-lzx3"
server_key: "90221420ad66581f642a0385afa40932"
server_key_iv: "1fe64176a9058fa37f07303f4879bfa2"
server_key_version: "encryptionkey@0"
I20260621 07:49:36.170418 14163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:49:36.179935 14163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:49:36.182446 14163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:49:36.182605 14163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:49:36.189045 14163 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.253:42889
I20260621 07:49:36.189061 14230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.253:42889 every 8 connection(s)
I20260621 07:49:36.189837 14163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-1/data/info.pb
I20260621 07:49:36.192144 14231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:49:36.195174 14231 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34803 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:49:36.196425 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 14163
I20260621 07:49:36.196537 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-1/wal/instance
I20260621 07:49:36.196811 771 external_mini_cluster.cc:1428] Setting key ba083e0a874c72354e0029af858e2318
Jun 21 07:49:36 dist-test-slave-lzx3 krb5kdc[14075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.253@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.192.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.252@KRBTEST.COM" created.
I20260621 07:49:36.208741 14159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:36.196648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45870 (local address 127.0.192.254:34803)
0621 07:49:36.196777 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:36.196780 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:36.197582 (+ 802us) server_negotiation.cc:407] Connection header received
0621 07:49:36.198540 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:36.198543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:36.198585 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:36.198648 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:49:36.200086 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.200713 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.202492 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.202718 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.202799 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:36.205261 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:49:36.205264 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:49:36.205266 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:49:36.205288 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:49:36.206652 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:36.207218 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:36.207222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:36.207224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:36.207294 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:36.207603 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:36.207607 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:36.207608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:36.207777 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:49:36.207878 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:36.208479 (+ 601us) server_negotiation.cc:299] Negotiation successful
0621 07:49:36.208629 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260621 07:49:36.209656 14233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:36.197034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:34803 (local address 127.0.0.1:45870)
0621 07:49:36.197458 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:36.197493 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 07:49:36.198315 (+ 822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:49:36.198833 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:49:36.198842 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:49:36.199315 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:49:36.199912 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:36.199932 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.200847 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.200852 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:36.202333 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:36.202347 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.202866 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.202870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:36.203048 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:36.203569 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:49:36.203590 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:49:36.205088 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:49:36.206757 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:36.206761 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:36.206770 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:36.207104 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:36.207378 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:36.207382 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:36.207384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:36.207499 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:36.207879 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:49:36.207885 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:49:36.208204 (+ 319us) client_negotiation.cc:769] Sending connection context
0621 07:49:36.208410 (+ 206us) client_negotiation.cc:240] Negotiation successful
0621 07:49:36.208742 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":144,"threads_started":1}
I20260621 07:49:36.210914 14231 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.192.253" port: 42889 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 07:49:36 dist-test-slave-lzx3 krb5kdc[14075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.253@KRBTEST.COM for kudu/127.0.192.253@KRBTEST.COM
I20260621 07:49:36.220182 14233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:36.211390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.253:42889 (local address 127.0.0.1:54142)
0621 07:49:36.211562 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:36.211579 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:49:36.211663 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:49:36.212046 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:49:36.212051 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:49:36.212217 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:49:36.212344 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:36.212353 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.213258 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.213263 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:36.214087 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:36.214101 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.214612 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.214615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:36.214682 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:36.215090 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:49:36.215107 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:49:36.216767 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:49:36.218728 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:36.218731 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:36.218733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:36.218913 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:36.219242 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:36.219244 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:36.219246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:36.219319 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:36.219724 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:49:36.219727 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:49:36.219826 (+ 99us) client_negotiation.cc:769] Sending connection context
0621 07:49:36.219934 (+ 108us) client_negotiation.cc:240] Negotiation successful
0621 07:49:36.220041 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260621 07:49:36.220360 14236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:36.211483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54142 (local address 127.0.192.253:42889)
0621 07:49:36.211743 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:36.211752 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:36.211768 (+ 16us) server_negotiation.cc:407] Connection header received
0621 07:49:36.211799 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:36.211802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:36.211923 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:36.212020 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:49:36.212458 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.213135 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.214222 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.214522 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.214590 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:36.216882 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:49:36.216890 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:49:36.216907 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:49:36.216924 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:49:36.218584 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:36.219019 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:36.219024 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:36.219026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:36.219118 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:36.219404 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:36.219407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:36.219408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:36.219613 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:49:36.219732 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:36.220017 (+ 285us) server_negotiation.cc:299] Negotiation successful
0621 07:49:36.220114 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":108,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/krb5kdc/kudu_127.0.192.252.keytab.
Entry for principal kudu/127.0.192.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/krb5kdc/kudu_127.0.192.252.keytab.
I20260621 07:49:36.221459 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.252:42621
--webserver_interface=127.0.192.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/krb5kdc/kudu_127.0.192.252.keytab
--principal=kudu/127.0.192.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:38749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:34803,127.0.192.253:42889,127.0.192.252:42621
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:49:36.222582 14231 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42621 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:49:36.223155 14233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:36.222973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:42621 (local address 127.0.0.1:51598)
0621 07:49:36.223059 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:36.223112 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.252:42621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260621 07:49:36.223309 14179 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.192.252:42621: connect: Connection refused (error 111)
I20260621 07:49:36.223973 14233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:36.223799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:42621 (local address 127.0.0.1:51610)
0621 07:49:36.223887 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:36.223924 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.252:42621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260621 07:49:36.224112 14231 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.192.252:42621: Network error: Client connection negotiation failed: client connection to 127.0.192.252:42621: connect: Connection refused (error 111)
I20260621 07:49:36.245023 14156 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.192.253" port: 42889 } attempt: 3
Jun 21 07:49:36 dist-test-slave-lzx3 krb5kdc[14075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for kudu/127.0.192.253@KRBTEST.COM
I20260621 07:49:36.253302 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:36.245468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.253:42889 (local address 127.0.0.1:54152)
0621 07:49:36.245588 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:36.245604 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:49:36.245688 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:49:36.245930 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:49:36.245933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:49:36.246102 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:49:36.246524 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:36.246530 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.247289 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.247294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:36.247792 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:36.247799 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.248792 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.248795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:36.248908 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:36.249379 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:49:36.249390 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:49:36.250478 (+ 1088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:49:36.251973 (+ 1495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:36.251977 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:36.251979 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:36.252156 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:36.252415 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:36.252418 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:36.252420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:36.252496 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:36.252848 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:49:36.252850 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:49:36.252940 (+ 90us) client_negotiation.cc:769] Sending connection context
0621 07:49:36.253037 (+ 97us) client_negotiation.cc:240] Negotiation successful
0621 07:49:36.253150 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260621 07:49:36.253302 14236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:36.245528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54152 (local address 127.0.192.253:42889)
0621 07:49:36.245648 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:36.245651 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:36.245675 (+ 24us) server_negotiation.cc:407] Connection header received
0621 07:49:36.245792 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:36.245795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:36.245838 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:36.245916 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:49:36.246648 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.247192 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.247946 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.248674 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.248781 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:36.250605 (+ 1824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:49:36.250609 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:49:36.250611 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:49:36.250650 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:49:36.251870 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:36.252254 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:36.252258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:36.252260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:36.252331 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:36.252598 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:36.252601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:36.252603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:36.252755 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:49:36.252855 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:36.253091 (+ 236us) server_negotiation.cc:299] Negotiation successful
0621 07:49:36.253187 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 07:49:36.253961 14156 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42621 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:49:36.254541 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:36.254327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:42621 (local address 127.0.0.1:51620)
0621 07:49:36.254419 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:36.254487 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.252:42621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260621 07:49:36.255080 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:36.254883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:42621 (local address 127.0.0.1:51628)
0621 07:49:36.254996 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:36.255033 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.252:42621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260621 07:49:36.255208 14156 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.192.252:42621: Network error: Client connection negotiation failed: client connection to 127.0.192.252:42621: connect: Connection refused (error 111)
I20260621 07:49:36.273329 14231 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42621 } attempt: 1
I20260621 07:49:36.273842 14233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:36.273634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:42621 (local address 127.0.0.1:51642)
0621 07:49:36.273747 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:36.273797 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.252:42621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260621 07:49:36.274417 14233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:36.274240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:42621 (local address 127.0.0.1:51646)
0621 07:49:36.274331 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:36.274370 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.252:42621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260621 07:49:36.274537 14231 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.192.252:42621: Network error: Client connection negotiation failed: client connection to 127.0.192.252:42621: connect: Connection refused (error 111)
I20260621 07:49:36.286356 14156 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42621 } attempt: 1
I20260621 07:49:36.286821 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:36.286653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:42621 (local address 127.0.0.1:51650)
0621 07:49:36.286741 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:36.286779 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.252:42621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260621 07:49:36.287366 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:36.287197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:42621 (local address 127.0.0.1:51662)
0621 07:49:36.287285 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:36.287327 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.252:42621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260621 07:49:36.287473 14156 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.192.252:42621: Network error: Client connection negotiation failed: client connection to 127.0.192.252:42621: connect: Connection refused (error 111)
W20260621 07:49:36.322557 14237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:49:36.322793 14237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:49:36.322875 14237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:49:36.326660 14237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:49:36.326736 14237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:49:36.326776 14237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:49:36.326830 14237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:49:36.326869 14237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:49:36.331473 14237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:38749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:34803,127.0.192.253:42889,127.0.192.252:42621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/krb5kdc/kudu_127.0.192.252.keytab
--principal=kudu/127.0.192.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.252:42621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.192.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:49:36.332340 14237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:49:36.333278 14237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:49:36.340788 14244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:49:36.340770 14243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:49:36.340870 14237 server_base.cc:1034] running on GCE node
W20260621 07:49:36.340765 14246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:49:36.341519 14237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:49:36.342360 14237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:49:36.342677 14231 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42621 } attempt: 2
I20260621 07:49:36.343210 14233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:36.343010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:42621 (local address 127.0.0.1:51674)
0621 07:49:36.343109 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:36.343152 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.252:42621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260621 07:49:36.343513 14237 hybrid_clock.cc:648] HybridClock initialized: now 1782028176343492 us; error 37 us; skew 500 ppm
I20260621 07:49:36.343806 14233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:36.343574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:42621 (local address 127.0.0.1:51676)
0621 07:49:36.343696 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:36.343741 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.252:42621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260621 07:49:36.343931 14231 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.192.252:42621: Network error: Client connection negotiation failed: client connection to 127.0.192.252:42621: connect: Connection refused (error 111)
Jun 21 07:49:36 dist-test-slave-lzx3 krb5kdc[14075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:49:36.346076 14237 init.cc:377] Logged in from keytab as kudu/127.0.192.252@KRBTEST.COM (short username kudu)
I20260621 07:49:36.346750 14237 webserver.cc:460] Webserver started at http://127.0.192.252:34977/ using document root <none> and password file <none>
I20260621 07:49:36.347288 14237 fs_manager.cc:359] Metadata directory not provided
I20260621 07:49:36.347345 14237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:49:36.347479 14237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:49:36.349264 14237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-2/data/instance:
uuid: "2be421a4d4ac4d83bbf3774369134c91"
format_stamp: "Formatted at 2026-06-21 07:49:36 on dist-test-slave-lzx3"
server_key: "43f05129376d221f27020547e641de7d"
server_key_iv: "c8b7d904f47951f1eda304605b08fa3b"
server_key_version: "encryptionkey@0"
I20260621 07:49:36.349694 14237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-2/wal/instance:
uuid: "2be421a4d4ac4d83bbf3774369134c91"
format_stamp: "Formatted at 2026-06-21 07:49:36 on dist-test-slave-lzx3"
server_key: "43f05129376d221f27020547e641de7d"
server_key_iv: "c8b7d904f47951f1eda304605b08fa3b"
server_key_version: "encryptionkey@0"
I20260621 07:49:36.352865 14237 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:49:36.354970 14253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:36.356019 14237 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:49:36.356149 14237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-2/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-2/wal
uuid: "2be421a4d4ac4d83bbf3774369134c91"
format_stamp: "Formatted at 2026-06-21 07:49:36 on dist-test-slave-lzx3"
server_key: "43f05129376d221f27020547e641de7d"
server_key_iv: "c8b7d904f47951f1eda304605b08fa3b"
server_key_version: "encryptionkey@0"
I20260621 07:49:36.356244 14237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:49:36.362607 14156 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42621 } attempt: 2
I20260621 07:49:36.363320 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:36.363097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:42621 (local address 127.0.0.1:51682)
0621 07:49:36.363214 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:36.363260 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.252:42621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260621 07:49:36.363956 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:36.363748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:42621 (local address 127.0.0.1:51684)
0621 07:49:36.363862 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:36.363910 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.252:42621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260621 07:49:36.364087 14156 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.192.252:42621: Network error: Client connection negotiation failed: client connection to 127.0.192.252:42621: connect: Connection refused (error 111)
I20260621 07:49:36.386309 14237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:49:36.389008 14237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:49:36.389197 14237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:49:36.395784 14237 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.252:42621
I20260621 07:49:36.395807 14305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.252:42621 every 8 connection(s)
I20260621 07:49:36.396674 14237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-2/data/info.pb
I20260621 07:49:36.399057 14306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:49:36.402154 14306 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34803 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:49:36.406345 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 14237
I20260621 07:49:36.406471 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782028028522580-771-0/minicluster-data/master-2/wal/instance
I20260621 07:49:36.406704 771 external_mini_cluster.cc:1428] Setting key 69da7b031d4708350d282f6dcc6bf457
Jun 21 07:49:36 dist-test-slave-lzx3 krb5kdc[14075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.252@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
Jun 21 07:49:36 dist-test-slave-lzx3 krb5kdc[14075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.192.252@KRBTEST.COM
I20260621 07:49:36.414393 14159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:36.403635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45874 (local address 127.0.192.254:34803)
0621 07:49:36.403736 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:36.403739 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:36.404423 (+ 684us) server_negotiation.cc:407] Connection header received
0621 07:49:36.405242 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:36.405244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:36.405285 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:36.405355 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:49:36.406375 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.406930 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.408353 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.408641 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.408748 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:36.410905 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:49:36.410909 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:49:36.410912 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:49:36.410936 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:49:36.412294 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:36.412814 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:36.412818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:36.412820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:36.412901 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:36.413298 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:36.413302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:36.413305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:36.413560 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:49:36.413677 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:36.414105 (+ 428us) server_negotiation.cc:299] Negotiation successful
0621 07:49:36.414217 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260621 07:49:36.414954 14308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:36.403945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:34803 (local address 127.0.0.1:45874)
0621 07:49:36.404297 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:36.404331 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0621 07:49:36.405076 (+ 745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:49:36.405488 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:49:36.405493 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:49:36.405784 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:49:36.406248 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:36.406267 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.407054 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.407056 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:36.408232 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:36.408239 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.408778 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.408782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:36.408944 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:36.409446 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:49:36.409459 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:49:36.410760 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:49:36.412396 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:36.412403 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:36.412415 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:36.412726 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:36.413019 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:36.413023 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:36.413025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:36.413196 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:36.413691 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:49:36.413696 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:49:36.413864 (+ 168us) client_negotiation.cc:769] Sending connection context
0621 07:49:36.414010 (+ 146us) client_negotiation.cc:240] Negotiation successful
0621 07:49:36.414232 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":99,"threads_started":1}
I20260621 07:49:36.416216 14306 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.192.253" port: 42889 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:49:36.416568 14310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:36.407247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51698 (local address 127.0.192.252:42621)
0621 07:49:36.407610 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:36.407620 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:36.407640 (+ 20us) server_negotiation.cc:407] Connection header received
0621 07:49:36.407675 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:36.407680 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:36.407848 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:36.408000 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:49:36.408665 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.409358 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.410155 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.410484 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.410558 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:36.412701 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:49:36.412709 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:49:36.412717 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:49:36.412736 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:49:36.414443 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:36.414866 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:36.414870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:36.414872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:36.414953 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:36.415571 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:36.415575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:36.415576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:36.415838 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:49:36.416022 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:36.416225 (+ 203us) server_negotiation.cc:299] Negotiation successful
0621 07:49:36.416352 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":118,"threads_started":1}
Jun 21 07:49:36 dist-test-slave-lzx3 krb5kdc[14075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.252@KRBTEST.COM for kudu/127.0.192.253@KRBTEST.COM
Jun 21 07:49:36 dist-test-slave-lzx3 krb5kdc[14075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:49:36.424080 14308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:36.416649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.253:42889 (local address 127.0.0.1:54154)
0621 07:49:36.416773 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:36.416785 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:49:36.416848 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:49:36.417526 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:49:36.417528 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:49:36.417655 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:49:36.417744 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:36.417750 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.418930 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.418932 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:36.419364 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:36.419370 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.419768 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.419771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:36.419854 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:36.420225 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:49:36.420243 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:49:36.421193 (+ 950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:49:36.422848 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:36.422852 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:36.422854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:36.423017 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:36.423273 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:36.423276 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:36.423277 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:36.423349 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:36.423674 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:49:36.423678 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:49:36.423753 (+ 75us) client_negotiation.cc:769] Sending connection context
0621 07:49:36.423840 (+ 87us) client_negotiation.cc:240] Negotiation successful
0621 07:49:36.423937 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260621 07:49:36.424122 14236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:36.417244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54154 (local address 127.0.192.253:42889)
0621 07:49:36.417354 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:36.417357 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:36.417367 (+ 10us) server_negotiation.cc:407] Connection header received
0621 07:49:36.417399 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:36.417401 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:36.417433 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:36.417516 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:49:36.418073 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.418825 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.419479 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.419676 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.419768 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:36.421308 (+ 1540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:49:36.421310 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:49:36.421313 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:49:36.421339 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:49:36.422756 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:36.423106 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:36.423109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:36.423111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:36.423186 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:36.423426 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:36.423430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:36.423432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:36.423577 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:49:36.423665 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:36.423896 (+ 231us) server_negotiation.cc:299] Negotiation successful
0621 07:49:36.424002 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260621 07:49:36.425187 14306 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42621 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:49:36.426592 14159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:36.418267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45876 (local address 127.0.192.254:34803)
0621 07:49:36.418361 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:36.418366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:36.418483 (+ 117us) server_negotiation.cc:407] Connection header received
0621 07:49:36.418595 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:36.418597 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:36.418629 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:36.418691 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:49:36.419323 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.419720 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.420381 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.420618 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.420726 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:36.423106 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:49:36.423110 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:49:36.423112 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:49:36.423134 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:49:36.424766 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:36.425313 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:36.425317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:36.425319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:36.425414 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:36.425753 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:36.425758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:36.425763 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:36.425991 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:49:36.426186 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:36.426347 (+ 161us) server_negotiation.cc:299] Negotiation successful
0621 07:49:36.426458 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
Jun 21 07:49:36 dist-test-slave-lzx3 krb5kdc[14075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.252@KRBTEST.COM for kudu/127.0.192.252@KRBTEST.COM
Jun 21 07:49:36 dist-test-slave-lzx3 krb5kdc[14075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.192.253@KRBTEST.COM
I20260621 07:49:36.433570 14310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:36.425612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51710 (local address 127.0.192.252:42621)
0621 07:49:36.425710 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:36.425715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:36.425748 (+ 33us) server_negotiation.cc:407] Connection header received
0621 07:49:36.425846 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:36.425850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:36.425888 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:36.425986 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:49:36.426318 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.426909 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.427668 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.427891 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.427997 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:36.429867 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:49:36.429872 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:49:36.429876 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:49:36.429905 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:49:36.431695 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:36.432146 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:36.432151 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:36.432155 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:36.432245 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:36.432549 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:36.432571 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:36.432573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:36.432795 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:49:36.432902 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:36.433188 (+ 286us) server_negotiation.cc:299] Negotiation successful
0621 07:49:36.433318 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":37}
I20260621 07:49:36.433457 14308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:36.425586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:42621 (local address 127.0.0.1:51710)
0621 07:49:36.425697 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:36.425711 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:49:36.425779 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:49:36.425995 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:49:36.425996 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:49:36.426134 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:49:36.426207 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:36.426213 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.427016 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.427019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:36.427489 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:36.427497 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.427994 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.427998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:36.428120 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:36.428536 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:49:36.428546 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:49:36.429708 (+ 1162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:49:36.431813 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:36.431818 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:36.431822 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:36.432041 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:36.432357 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:36.432363 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:36.432367 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:36.432453 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:36.432886 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:49:36.432889 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:49:36.432975 (+ 86us) client_negotiation.cc:769] Sending connection context
0621 07:49:36.433121 (+ 146us) client_negotiation.cc:240] Negotiation successful
0621 07:49:36.433238 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":17}
I20260621 07:49:36.435057 14231 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42621 } attempt: 3
I20260621 07:49:36.436751 14236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:36.427538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54156 (local address 127.0.192.253:42889)
0621 07:49:36.427625 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:36.427628 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:36.427684 (+ 56us) server_negotiation.cc:407] Connection header received
0621 07:49:36.427822 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:36.427825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:36.427888 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:36.427979 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:49:36.428749 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.429470 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.430272 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.430572 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.430686 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:36.432851 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:49:36.432856 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:49:36.432858 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:49:36.432882 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:49:36.434380 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:36.434858 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:36.434861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:36.434863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:36.434938 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:36.435544 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:36.435551 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:36.435553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:36.435785 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:49:36.435986 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:36.436351 (+ 365us) server_negotiation.cc:299] Negotiation successful
0621 07:49:36.436464 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260621 07:49:36.437093 14306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2be421a4d4ac4d83bbf3774369134c91: Bootstrap starting.
I20260621 07:49:36.439723 14306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2be421a4d4ac4d83bbf3774369134c91: Neither blocks nor log segments found. Creating new log.
I20260621 07:49:36.439980 771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
Jun 21 07:49:36 dist-test-slave-lzx3 krb5kdc[14075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.253@KRBTEST.COM for kudu/127.0.192.252@KRBTEST.COM
I20260621 07:49:36.440688 14306 log.cc:824] T 00000000000000000000000000000000 P 2be421a4d4ac4d83bbf3774369134c91: Log is configured to *not* fsync() on all Append() calls
I20260621 07:49:36.443065 14306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2be421a4d4ac4d83bbf3774369134c91: No bootstrap required, opened a new log
I20260621 07:49:36.444334 14233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:36.435421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:42621 (local address 127.0.0.1:51718)
0621 07:49:36.435648 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:36.435662 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:49:36.435742 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:49:36.436175 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:49:36.436180 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:49:36.436330 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:49:36.436410 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:36.436416 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.437359 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.437362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:36.438082 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:36.438092 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.438673 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.438677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:36.438765 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:36.439225 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:49:36.439241 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:49:36.440772 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:49:36.442766 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:36.442769 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:36.442771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:36.443002 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:36.443314 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:36.443317 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:36.443319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:36.443416 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:36.443861 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:49:36.443864 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:49:36.443949 (+ 85us) client_negotiation.cc:769] Sending connection context
0621 07:49:36.444048 (+ 99us) client_negotiation.cc:240] Negotiation successful
0621 07:49:36.444184 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152}
I20260621 07:49:36.444811 14310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:36.435881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51718 (local address 127.0.192.252:42621)
0621 07:49:36.435967 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:36.435971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:36.435981 (+ 10us) server_negotiation.cc:407] Connection header received
0621 07:49:36.436009 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:36.436011 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:36.436043 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:36.436511 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:49:36.436613 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.437240 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.438247 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.438559 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.438653 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:36.440912 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:49:36.440915 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:49:36.440918 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:49:36.440939 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:49:36.442641 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:36.443115 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:36.443118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:36.443120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:36.443212 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:36.443518 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:36.443521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:36.443522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:36.443730 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:49:36.443817 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:36.444505 (+ 688us) server_negotiation.cc:299] Negotiation successful
0621 07:49:36.444656 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260621 07:49:36.447979 14306 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2be421a4d4ac4d83bbf3774369134c91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359c9c7393414440b566e6e1dcaa6f2c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34803 } } peers { permanent_uuid: "e21c6dc532ca420b9d9a333c9bc6f5c8" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 42889 } } peers { permanent_uuid: "2be421a4d4ac4d83bbf3774369134c91" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42621 } }
I20260621 07:49:36.448205 14231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e21c6dc532ca420b9d9a333c9bc6f5c8: Bootstrap starting.
I20260621 07:49:36.448276 14306 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2be421a4d4ac4d83bbf3774369134c91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:49:36.448333 14306 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2be421a4d4ac4d83bbf3774369134c91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2be421a4d4ac4d83bbf3774369134c91, State: Initialized, Role: FOLLOWER
I20260621 07:49:36.448840 14306 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2be421a4d4ac4d83bbf3774369134c91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359c9c7393414440b566e6e1dcaa6f2c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34803 } } peers { permanent_uuid: "e21c6dc532ca420b9d9a333c9bc6f5c8" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 42889 } } peers { permanent_uuid: "2be421a4d4ac4d83bbf3774369134c91" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42621 } }
I20260621 07:49:36.450206 14306 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2be421a4d4ac4d83bbf3774369134c91 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:49:36.449465 14320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2be421a4d4ac4d83bbf3774369134c91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359c9c7393414440b566e6e1dcaa6f2c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34803 } } peers { permanent_uuid: "e21c6dc532ca420b9d9a333c9bc6f5c8" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 42889 } } peers { permanent_uuid: "2be421a4d4ac4d83bbf3774369134c91" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42621 } } }
I20260621 07:49:36.451000 14320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2be421a4d4ac4d83bbf3774369134c91 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:49:36.452124 14231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e21c6dc532ca420b9d9a333c9bc6f5c8: Neither blocks nor log segments found. Creating new log.
I20260621 07:49:36.453409 14231 log.cc:824] T 00000000000000000000000000000000 P e21c6dc532ca420b9d9a333c9bc6f5c8: Log is configured to *not* fsync() on all Append() calls
I20260621 07:49:36.455283 14318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:36.443586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51722 (local address 127.0.192.252:42621)
0621 07:49:36.444792 (+ 1206us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:36.444803 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:36.445218 (+ 415us) server_negotiation.cc:407] Connection header received
0621 07:49:36.445252 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:36.445255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:36.445302 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:36.445375 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:49:36.446229 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.446949 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.448413 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.448774 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.448871 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:36.450554 (+ 1683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:49:36.450559 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:49:36.450562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:49:36.450588 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:49:36.452531 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:36.453747 (+ 1216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:36.453750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:36.453752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:36.453825 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:36.454092 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:36.454095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:36.454097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:36.454318 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:49:36.454439 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:36.454993 (+ 554us) server_negotiation.cc:299] Negotiation successful
0621 07:49:36.455094 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1119,"thread_start_us":99,"threads_started":1}
W20260621 07:49:36.455319 14331 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2be421a4d4ac4d83bbf3774369134c91: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:49:36.455367 14331 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:49:36.456915 14159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:36.442547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45890 (local address 127.0.192.254:34803)
0621 07:49:36.442674 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:36.442681 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:36.443070 (+ 389us) server_negotiation.cc:407] Connection header received
0621 07:49:36.443274 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:36.443278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:36.443337 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:36.443395 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:49:36.444697 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.445393 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.446552 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.446870 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.446962 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:36.452521 (+ 5559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:49:36.452525 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:49:36.452528 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:49:36.452602 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:49:36.454979 (+ 2377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:36.455575 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:36.455579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:36.455581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:36.455665 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:36.455978 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:36.455981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:36.455983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:36.456187 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:49:36.456288 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:36.456652 (+ 364us) server_negotiation.cc:299] Negotiation successful
0621 07:49:36.456763 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 07:49:36.454207 14237 master_runner.cc:186] Error getting master registration for 127.0.192.254:34803: OK, instance_id { permanent_uuid: "359c9c7393414440b566e6e1dcaa6f2c" instance_seqno: 1782028176006940 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 07:49:36.458873 14237 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 07:49:36.459686 14236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:36.447316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54168 (local address 127.0.192.253:42889)
0621 07:49:36.447602 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:36.447606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:36.447618 (+ 12us) server_negotiation.cc:407] Connection header received
0621 07:49:36.447648 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:36.447663 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:36.447699 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:36.447778 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:49:36.448725 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.449380 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.450529 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.450812 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.450900 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:36.456535 (+ 5635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:49:36.456538 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:49:36.456541 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:49:36.456595 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:49:36.458138 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:36.458552 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:36.458555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:36.458557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:36.458639 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:36.458890 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:36.458895 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:36.458897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:36.459086 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:49:36.459222 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:36.459452 (+ 230us) server_negotiation.cc:299] Negotiation successful
0621 07:49:36.459554 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225}
I20260621 07:49:36.462252 14231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e21c6dc532ca420b9d9a333c9bc6f5c8: No bootstrap required, opened a new log
I20260621 07:49:36.463233 14156 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42621 } attempt: 3
I20260621 07:49:36.466117 14231 raft_consensus.cc:348] T 00000000000000000000000000000000 P e21c6dc532ca420b9d9a333c9bc6f5c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359c9c7393414440b566e6e1dcaa6f2c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34803 } } peers { permanent_uuid: "e21c6dc532ca420b9d9a333c9bc6f5c8" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 42889 } } peers { permanent_uuid: "2be421a4d4ac4d83bbf3774369134c91" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42621 } }
I20260621 07:49:36.466390 14231 raft_consensus.cc:374] T 00000000000000000000000000000000 P e21c6dc532ca420b9d9a333c9bc6f5c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:49:36.466483 14231 raft_consensus.cc:729] T 00000000000000000000000000000000 P e21c6dc532ca420b9d9a333c9bc6f5c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e21c6dc532ca420b9d9a333c9bc6f5c8, State: Initialized, Role: FOLLOWER
I20260621 07:49:36.467244 14231 consensus_queue.cc:260] T 00000000000000000000000000000000 P e21c6dc532ca420b9d9a333c9bc6f5c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359c9c7393414440b566e6e1dcaa6f2c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34803 } } peers { permanent_uuid: "e21c6dc532ca420b9d9a333c9bc6f5c8" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 42889 } } peers { permanent_uuid: "2be421a4d4ac4d83bbf3774369134c91" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42621 } }
I20260621 07:49:36.467778 14334 sys_catalog.cc:455] T 00000000000000000000000000000000 P e21c6dc532ca420b9d9a333c9bc6f5c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359c9c7393414440b566e6e1dcaa6f2c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34803 } } peers { permanent_uuid: "e21c6dc532ca420b9d9a333c9bc6f5c8" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 42889 } } peers { permanent_uuid: "2be421a4d4ac4d83bbf3774369134c91" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42621 } } }
I20260621 07:49:36.467967 14334 sys_catalog.cc:458] T 00000000000000000000000000000000 P e21c6dc532ca420b9d9a333c9bc6f5c8 [sys.catalog]: This master's current role is: FOLLOWER
Jun 21 07:49:36 dist-test-slave-lzx3 krb5kdc[14075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for kudu/127.0.192.252@KRBTEST.COM
I20260621 07:49:36.468582 14231 sys_catalog.cc:564] T 00000000000000000000000000000000 P e21c6dc532ca420b9d9a333c9bc6f5c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:49:36.472230 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:36.463586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:42621 (local address 127.0.0.1:51730)
0621 07:49:36.463797 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:36.463810 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:49:36.463880 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:49:36.464296 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:49:36.464299 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:49:36.464441 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:49:36.464501 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:36.464506 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.465276 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.465281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:36.465773 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:36.465780 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.466388 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.466394 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:36.466471 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:36.467165 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:49:36.467180 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:49:36.468521 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:49:36.470876 (+ 2355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:36.470879 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:36.470883 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:36.471086 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:36.471355 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:36.471358 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:36.471360 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:36.471466 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:36.471847 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:49:36.471849 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:49:36.471926 (+ 77us) client_negotiation.cc:769] Sending connection context
0621 07:49:36.471998 (+ 72us) client_negotiation.cc:240] Negotiation successful
0621 07:49:36.472105 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149}
I20260621 07:49:36.472955 14318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:36.464007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51730 (local address 127.0.192.252:42621)
0621 07:49:36.464129 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:36.464133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:36.464153 (+ 20us) server_negotiation.cc:407] Connection header received
0621 07:49:36.464177 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:36.464179 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:36.464205 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:36.464282 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:49:36.464634 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.465177 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.465888 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:36.466178 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:36.466280 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:36.468696 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:49:36.468699 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:49:36.468704 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:49:36.468728 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:49:36.470758 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:36.471193 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:36.471195 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:36.471197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:36.471266 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:36.471567 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:36.471569 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:36.471571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:36.471747 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:49:36.472686 (+ 939us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:36.472762 (+ 76us) server_negotiation.cc:299] Negotiation successful
0621 07:49:36.472848 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260621 07:49:36.474912 14345 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e21c6dc532ca420b9d9a333c9bc6f5c8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:49:36.475023 14345 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:49:36.471984 14163 master_runner.cc:186] Error getting master registration for 127.0.192.254:34803: OK, instance_id { permanent_uuid: "359c9c7393414440b566e6e1dcaa6f2c" instance_seqno: 1782028176006940 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 07:49:36.475591 14163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 07:49:36.475616 14156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 359c9c7393414440b566e6e1dcaa6f2c: Bootstrap starting.
I20260621 07:49:36.478091 14156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 359c9c7393414440b566e6e1dcaa6f2c: Neither blocks nor log segments found. Creating new log.
I20260621 07:49:36.479045 14156 log.cc:824] T 00000000000000000000000000000000 P 359c9c7393414440b566e6e1dcaa6f2c: Log is configured to *not* fsync() on all Append() calls
I20260621 07:49:36.480912 14156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 359c9c7393414440b566e6e1dcaa6f2c: No bootstrap required, opened a new log
I20260621 07:49:36.484401 14156 raft_consensus.cc:348] T 00000000000000000000000000000000 P 359c9c7393414440b566e6e1dcaa6f2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359c9c7393414440b566e6e1dcaa6f2c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34803 } } peers { permanent_uuid: "e21c6dc532ca420b9d9a333c9bc6f5c8" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 42889 } } peers { permanent_uuid: "2be421a4d4ac4d83bbf3774369134c91" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42621 } }
I20260621 07:49:36.484617 14156 raft_consensus.cc:374] T 00000000000000000000000000000000 P 359c9c7393414440b566e6e1dcaa6f2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:49:36.484676 14156 raft_consensus.cc:729] T 00000000000000000000000000000000 P 359c9c7393414440b566e6e1dcaa6f2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 359c9c7393414440b566e6e1dcaa6f2c, State: Initialized, Role: FOLLOWER
I20260621 07:49:36.485209 14156 consensus_queue.cc:260] T 00000000000000000000000000000000 P 359c9c7393414440b566e6e1dcaa6f2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359c9c7393414440b566e6e1dcaa6f2c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34803 } } peers { permanent_uuid: "e21c6dc532ca420b9d9a333c9bc6f5c8" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 42889 } } peers { permanent_uuid: "2be421a4d4ac4d83bbf3774369134c91" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42621 } }
I20260621 07:49:36.486276 14348 raft_consensus.cc:482] T 00000000000000000000000000000000 P 359c9c7393414440b566e6e1dcaa6f2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:49:36.486451 14348 raft_consensus.cc:504] T 00000000000000000000000000000000 P 359c9c7393414440b566e6e1dcaa6f2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359c9c7393414440b566e6e1dcaa6f2c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34803 } } peers { permanent_uuid: "e21c6dc532ca420b9d9a333c9bc6f5c8" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 42889 } } peers { permanent_uuid: "2be421a4d4ac4d83bbf3774369134c91" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42621 } }
I20260621 07:49:36.486515 14156 sys_catalog.cc:564] T 00000000000000000000000000000000 P 359c9c7393414440b566e6e1dcaa6f2c [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:49:36.487602 14348 leader_election.cc:288] T 00000000000000000000000000000000 P 359c9c7393414440b566e6e1dcaa6f2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e21c6dc532ca420b9d9a333c9bc6f5c8 (127.0.192.253:42889), 2be421a4d4ac4d83bbf3774369134c91 (127.0.192.252:42621)
I20260621 07:49:36.487783 14348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 359c9c7393414440b566e6e1dcaa6f2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359c9c7393414440b566e6e1dcaa6f2c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34803 } } peers { permanent_uuid: "e21c6dc532ca420b9d9a333c9bc6f5c8" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 42889 } } peers { permanent_uuid: "2be421a4d4ac4d83bbf3774369134c91" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42621 } } }
I20260621 07:49:36.487926 14348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 359c9c7393414440b566e6e1dcaa6f2c [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:49:36.488196 14205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "359c9c7393414440b566e6e1dcaa6f2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e21c6dc532ca420b9d9a333c9bc6f5c8" is_pre_election: true
I20260621 07:49:36.488278 14280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "359c9c7393414440b566e6e1dcaa6f2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2be421a4d4ac4d83bbf3774369134c91" is_pre_election: true
I20260621 07:49:36.488451 14205 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e21c6dc532ca420b9d9a333c9bc6f5c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 359c9c7393414440b566e6e1dcaa6f2c in term 0.
I20260621 07:49:36.488539 14280 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2be421a4d4ac4d83bbf3774369134c91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 359c9c7393414440b566e6e1dcaa6f2c in term 0.
I20260621 07:49:36.489068 14104 leader_election.cc:302] T 00000000000000000000000000000000 P 359c9c7393414440b566e6e1dcaa6f2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2be421a4d4ac4d83bbf3774369134c91, 359c9c7393414440b566e6e1dcaa6f2c; no voters:
I20260621 07:49:36.489526 14348 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 359c9c7393414440b566e6e1dcaa6f2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:49:36.489660 14348 raft_consensus.cc:482] T 00000000000000000000000000000000 P 359c9c7393414440b566e6e1dcaa6f2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:49:36.489749 14348 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 359c9c7393414440b566e6e1dcaa6f2c [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:49:36.490877 14348 raft_consensus.cc:504] T 00000000000000000000000000000000 P 359c9c7393414440b566e6e1dcaa6f2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359c9c7393414440b566e6e1dcaa6f2c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34803 } } peers { permanent_uuid: "e21c6dc532ca420b9d9a333c9bc6f5c8" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 42889 } } peers { permanent_uuid: "2be421a4d4ac4d83bbf3774369134c91" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42621 } }
I20260621 07:49:36.491425 14205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "359c9c7393414440b566e6e1dcaa6f2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e21c6dc532ca420b9d9a333c9bc6f5c8"
I20260621 07:49:36.491544 14205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e21c6dc532ca420b9d9a333c9bc6f5c8 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:49:36.492610 14205 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e21c6dc532ca420b9d9a333c9bc6f5c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 359c9c7393414440b566e6e1dcaa6f2c in term 1.
I20260621 07:49:36.492950 14104 leader_election.cc:302] T 00000000000000000000000000000000 P 359c9c7393414440b566e6e1dcaa6f2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 359c9c7393414440b566e6e1dcaa6f2c, e21c6dc532ca420b9d9a333c9bc6f5c8; no voters:
I20260621 07:49:36.496692 14349 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 359c9c7393414440b566e6e1dcaa6f2c [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:49:36.496913 14349 raft_consensus.cc:686] T 00000000000000000000000000000000 P 359c9c7393414440b566e6e1dcaa6f2c [term 1 LEADER]: Becoming Leader. State: Replica: 359c9c7393414440b566e6e1dcaa6f2c, State: Running, Role: LEADER
I20260621 07:49:36.497217 14349 consensus_queue.cc:237] T 00000000000000000000000000000000 P 359c9c7393414440b566e6e1dcaa6f2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359c9c7393414440b566e6e1dcaa6f2c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34803 } } peers { permanent_uuid: "e21c6dc532ca420b9d9a333c9bc6f5c8" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 42889 } } peers { permanent_uuid: "2be421a4d4ac4d83bbf3774369134c91" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42621 } }
I20260621 07:49:36.497853 14348 leader_election.cc:288] T 00000000000000000000000000000000 P 359c9c7393414440b566e6e1dcaa6f2c [CANDIDATE]: Term 1 election: Requested vote from peers e21c6dc532ca420b9d9a333c9bc6f5c8 (127.0.192.253:42889), 2be421a4d4ac4d83bbf3774369134c91 (127.0.192.252:42621)
I20260621 07:49:36.498697 14280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "359c9c7393414440b566e6e1dcaa6f2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2be421a4d4ac4d83bbf3774369134c91"
I20260621 07:49:36.498801 14280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2be421a4d4ac4d83bbf3774369134c91 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:49:36.499348 14348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 359c9c7393414440b566e6e1dcaa6f2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 359c9c7393414440b566e6e1dcaa6f2c. Latest consensus state: current_term: 1 leader_uuid: "359c9c7393414440b566e6e1dcaa6f2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359c9c7393414440b566e6e1dcaa6f2c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34803 } } peers { permanent_uuid: "e21c6dc532ca420b9d9a333c9bc6f5c8" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 42889 } } peers { permanent_uuid: "2be421a4d4ac4d83bbf3774369134c91" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42621 } } }
I20260621 07:49:36.499450 14348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 359c9c7393414440b566e6e1dcaa6f2c [sys.catalog]: This master's current role is: LEADER
I20260621 07:49:36.499899 14280 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2be421a4d4ac4d83bbf3774369134c91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 359c9c7393414440b566e6e1dcaa6f2c in term 1.
I20260621 07:49:36.500177 14088 master_runner.cc:232] Existing masters have differing terms: 1 vs 0
I20260621 07:49:36.500272 14088 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260621 07:49:36.502600 14360 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 359c9c7393414440b566e6e1dcaa6f2c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:49:36.502683 14360 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:49:36.502785 14363 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:49:36.503559 14363 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260621 07:49:36.503808 14109 tablet.cc:2369] T 00000000000000000000000000000000 P 359c9c7393414440b566e6e1dcaa6f2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:49:36.508893 14280 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2be421a4d4ac4d83bbf3774369134c91 [term 1 FOLLOWER]: Refusing update from remote peer 359c9c7393414440b566e6e1dcaa6f2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:49:36.509545 14361 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 359c9c7393414440b566e6e1dcaa6f2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be421a4d4ac4d83bbf3774369134c91" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:49:36.513032 14320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2be421a4d4ac4d83bbf3774369134c91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 359c9c7393414440b566e6e1dcaa6f2c. Latest consensus state: current_term: 1 leader_uuid: "359c9c7393414440b566e6e1dcaa6f2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359c9c7393414440b566e6e1dcaa6f2c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34803 } } peers { permanent_uuid: "e21c6dc532ca420b9d9a333c9bc6f5c8" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 42889 } } peers { permanent_uuid: "2be421a4d4ac4d83bbf3774369134c91" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42621 } } }
I20260621 07:49:36.513130 14320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2be421a4d4ac4d83bbf3774369134c91 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:49:36.514000 14205 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e21c6dc532ca420b9d9a333c9bc6f5c8 [term 1 FOLLOWER]: Refusing update from remote peer 359c9c7393414440b566e6e1dcaa6f2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:49:36.514520 14361 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 359c9c7393414440b566e6e1dcaa6f2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e21c6dc532ca420b9d9a333c9bc6f5c8" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 42889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:49:36.517407 14334 sys_catalog.cc:455] T 00000000000000000000000000000000 P e21c6dc532ca420b9d9a333c9bc6f5c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 359c9c7393414440b566e6e1dcaa6f2c. Latest consensus state: current_term: 1 leader_uuid: "359c9c7393414440b566e6e1dcaa6f2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359c9c7393414440b566e6e1dcaa6f2c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34803 } } peers { permanent_uuid: "e21c6dc532ca420b9d9a333c9bc6f5c8" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 42889 } } peers { permanent_uuid: "2be421a4d4ac4d83bbf3774369134c91" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42621 } } }
I20260621 07:49:36.517496 14334 sys_catalog.cc:458] T 00000000000000000000000000000000 P e21c6dc532ca420b9d9a333c9bc6f5c8 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:49:36.521013 14361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 359c9c7393414440b566e6e1dcaa6f2c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "359c9c7393414440b566e6e1dcaa6f2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359c9c7393414440b566e6e1dcaa6f2c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34803 } } peers { permanent_uuid: "e21c6dc532ca420b9d9a333c9bc6f5c8" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 42889 } } peers { permanent_uuid: "2be421a4d4ac4d83bbf3774369134c91" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42621 } } }
I20260621 07:49:36.521100 14361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 359c9c7393414440b566e6e1dcaa6f2c [sys.catalog]: This master's current role is: LEADER
I20260621 07:49:36.522159 14334 sys_catalog.cc:455] T 00000000000000000000000000000000 P e21c6dc532ca420b9d9a333c9bc6f5c8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "359c9c7393414440b566e6e1dcaa6f2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359c9c7393414440b566e6e1dcaa6f2c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34803 } } peers { permanent_uuid: "e21c6dc532ca420b9d9a333c9bc6f5c8" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 42889 } } peers { permanent_uuid: "2be421a4d4ac4d83bbf3774369134c91" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42621 } } }
I20260621 07:49:36.522243 14334 sys_catalog.cc:458] T 00000000000000000000000000000000 P e21c6dc532ca420b9d9a333c9bc6f5c8 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:49:36.522573 14365 mvcc.cc:205] Tried to move back new op lower bound from 7299187410974138368 to 7299187410936963072. Current Snapshot: MvccSnapshot[applied={T|T < 7299187410974138368}]
I20260621 07:49:36.523159 14368 mvcc.cc:205] Tried to move back new op lower bound from 7299187410974138368 to 7299187410936963072. Current Snapshot: MvccSnapshot[applied={T|T < 7299187410974138368}]
I20260621 07:49:36.524739 14349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 359c9c7393414440b566e6e1dcaa6f2c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "359c9c7393414440b566e6e1dcaa6f2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359c9c7393414440b566e6e1dcaa6f2c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34803 } } peers { permanent_uuid: "e21c6dc532ca420b9d9a333c9bc6f5c8" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 42889 } } peers { permanent_uuid: "2be421a4d4ac4d83bbf3774369134c91" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42621 } } }
I20260621 07:49:36.524876 14349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 359c9c7393414440b566e6e1dcaa6f2c [sys.catalog]: This master's current role is: LEADER
I20260621 07:49:36.525688 14320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2be421a4d4ac4d83bbf3774369134c91 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "359c9c7393414440b566e6e1dcaa6f2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359c9c7393414440b566e6e1dcaa6f2c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34803 } } peers { permanent_uuid: "e21c6dc532ca420b9d9a333c9bc6f5c8" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 42889 } } peers { permanent_uuid: "2be421a4d4ac4d83bbf3774369134c91" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42621 } } }
I20260621 07:49:36.525774 14320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2be421a4d4ac4d83bbf3774369134c91 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:49:36.530113 14366 mvcc.cc:205] Tried to move back new op lower bound from 7299187410974138368 to 7299187410936963072. Current Snapshot: MvccSnapshot[applied={T|T < 7299187410974138368}]
I20260621 07:49:36.530507 14363 catalog_manager.cc:1347] Generated new cluster ID: 0c9b05778e3e4645bc717b6b09bbdd6f
I20260621 07:49:36.530555 14363 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:49:36.551046 14363 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:49:36.552711 14363 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:49:36.567576 14363 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 359c9c7393414440b566e6e1dcaa6f2c: Generated new TSK 0
I20260621 07:49:36.568392 14363 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:49:37.456436 14331 catalog_manager.cc:1259] Loaded cluster ID: 0c9b05778e3e4645bc717b6b09bbdd6f
I20260621 07:49:37.456528 14331 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2be421a4d4ac4d83bbf3774369134c91: loading cluster ID for follower catalog manager: success
I20260621 07:49:37.458171 14331 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2be421a4d4ac4d83bbf3774369134c91: acquiring CA information for follower catalog manager: success
I20260621 07:49:37.459388 14331 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2be421a4d4ac4d83bbf3774369134c91: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 07:49:37.475986 14345 catalog_manager.cc:1259] Loaded cluster ID: 0c9b05778e3e4645bc717b6b09bbdd6f
I20260621 07:49:37.476037 14345 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e21c6dc532ca420b9d9a333c9bc6f5c8: loading cluster ID for follower catalog manager: success
I20260621 07:49:37.477545 14345 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e21c6dc532ca420b9d9a333c9bc6f5c8: acquiring CA information for follower catalog manager: success
I20260621 07:49:37.478561 14345 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e21c6dc532ca420b9d9a333c9bc6f5c8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 07:49:37.631085 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 14088
I20260621 07:49:37.637040 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 14163
I20260621 07:49:37.642643 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 14237
2026-06-21T07:49:37Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2873 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028028522580-771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:49:37 dist-test-slave-lzx3 krb5kdc[14381](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:49:37 dist-test-slave-lzx3 krb5kdc[14381](info): set up 2 sockets
Jun 21 07:49:37 dist-test-slave-lzx3 krb5kdc[14381](info): commencing operation
krb5kdc: starting...
W20260621 07:49:38.620703 771 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028028522580-771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028028522580-771-0/krb5kdc/test-user.keytab.
Jun 21 07:49:38 dist-test-slave-lzx3 krb5kdc[14381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:49:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:49:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.192.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
I20260621 07:49:38.725144 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:45645
--webserver_interface=127.0.192.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:46479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:45645
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:49:38.824638 14394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:49:38.824839 14394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:49:38.824882 14394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:49:38.828586 14394 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:49:38.828675 14394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:49:38.828722 14394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:49:38.828756 14394 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:49:38.828785 14394 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:49:38.833272 14394 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:46479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:45645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:45645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:49:38.834010 14394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:49:38.834738 14394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:49:38.842145 14400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:49:38.842236 14402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:49:38.842247 14399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:49:38.842279 14394 server_base.cc:1034] running on GCE node
I20260621 07:49:38.842862 14394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:49:38.843712 14394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:49:38.844887 14394 hybrid_clock.cc:648] HybridClock initialized: now 1782028178844873 us; error 30 us; skew 500 ppm
Jun 21 07:49:38 dist-test-slave-lzx3 krb5kdc[14381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:49:38.847296 14394 init.cc:377] Logged in from keytab as kudu/127.0.192.254@KRBTEST.COM (short username kudu)
I20260621 07:49:38.848052 14394 webserver.cc:460] Webserver started at http://127.0.192.254:45161/ using document root <none> and password file <none>
I20260621 07:49:38.848521 14394 fs_manager.cc:359] Metadata directory not provided
I20260621 07:49:38.848625 14394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:49:38.848778 14394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:49:38.850479 14394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "02a95480a8ca4c6cb131b248b6f5de92"
format_stamp: "Formatted at 2026-06-21 07:49:38 on dist-test-slave-lzx3"
server_key: "1ecdb0c2ee43daf306bd0e6813e926b7"
server_key_iv: "25c3f3d9e18ff169474da0297eb481fe"
server_key_version: "encryptionkey@0"
I20260621 07:49:38.850965 14394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "02a95480a8ca4c6cb131b248b6f5de92"
format_stamp: "Formatted at 2026-06-21 07:49:38 on dist-test-slave-lzx3"
server_key: "1ecdb0c2ee43daf306bd0e6813e926b7"
server_key_iv: "25c3f3d9e18ff169474da0297eb481fe"
server_key_version: "encryptionkey@0"
I20260621 07:49:38.854112 14394 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260621 07:49:38.855983 14409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:38.857108 14394 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:49:38.857225 14394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "02a95480a8ca4c6cb131b248b6f5de92"
format_stamp: "Formatted at 2026-06-21 07:49:38 on dist-test-slave-lzx3"
server_key: "1ecdb0c2ee43daf306bd0e6813e926b7"
server_key_iv: "25c3f3d9e18ff169474da0297eb481fe"
server_key_version: "encryptionkey@0"
I20260621 07:49:38.857318 14394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:49:38.879596 14394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:49:38.881759 14394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:49:38.881927 14394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:49:38.887671 14394 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:45645
I20260621 07:49:38.887786 14461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:45645 every 8 connection(s)
I20260621 07:49:38.888515 14394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:49:38.889581 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 14394
I20260621 07:49:38.889696 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:49:38.889919 771 external_mini_cluster.cc:1428] Setting key 34e79ae8c469f0d92c97244239c30c9d
I20260621 07:49:38.891278 14462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:49:38.895673 14462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02a95480a8ca4c6cb131b248b6f5de92: Bootstrap starting.
Jun 21 07:49:38 dist-test-slave-lzx3 krb5kdc[14381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:49:38.897819 14462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02a95480a8ca4c6cb131b248b6f5de92: Neither blocks nor log segments found. Creating new log.
I20260621 07:49:38.898661 14462 log.cc:824] T 00000000000000000000000000000000 P 02a95480a8ca4c6cb131b248b6f5de92: Log is configured to *not* fsync() on all Append() calls
I20260621 07:49:38.900111 14462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02a95480a8ca4c6cb131b248b6f5de92: No bootstrap required, opened a new log
I20260621 07:49:38.901661 14465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:38.891005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60108 (local address 127.0.192.254:45645)
0621 07:49:38.891443 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:38.891457 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:38.891486 (+ 29us) server_negotiation.cc:407] Connection header received
0621 07:49:38.891989 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:38.892002 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:38.892260 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:38.892553 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:49:38.893276 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:38.894082 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:38.894826 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:38.895207 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:38.895361 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:38.897777 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:49:38.897781 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:49:38.897792 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:49:38.897819 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:49:38.899295 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:38.899682 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:38.899687 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:38.899690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:38.899781 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:38.900032 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:38.900036 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:38.900038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:38.900421 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:49:38.900625 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:38.900875 (+ 250us) server_negotiation.cc:299] Negotiation successful
0621 07:49:38.901111 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":135,"threads_started":1}
I20260621 07:49:38.903276 14462 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02a95480a8ca4c6cb131b248b6f5de92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02a95480a8ca4c6cb131b248b6f5de92" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45645 } }
I20260621 07:49:38.903555 14462 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02a95480a8ca4c6cb131b248b6f5de92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:49:38.903648 14462 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02a95480a8ca4c6cb131b248b6f5de92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02a95480a8ca4c6cb131b248b6f5de92, State: Initialized, Role: FOLLOWER
I20260621 07:49:38.904124 771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:49:38.904060 14462 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02a95480a8ca4c6cb131b248b6f5de92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02a95480a8ca4c6cb131b248b6f5de92" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45645 } }
I20260621 07:49:38.904325 14462 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02a95480a8ca4c6cb131b248b6f5de92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:49:38.904386 14462 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02a95480a8ca4c6cb131b248b6f5de92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:49:38.904472 14462 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02a95480a8ca4c6cb131b248b6f5de92 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:49:38.905390 14462 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02a95480a8ca4c6cb131b248b6f5de92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02a95480a8ca4c6cb131b248b6f5de92" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45645 } }
I20260621 07:49:38.905690 14462 leader_election.cc:302] T 00000000000000000000000000000000 P 02a95480a8ca4c6cb131b248b6f5de92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02a95480a8ca4c6cb131b248b6f5de92; no voters:
I20260621 07:49:38.905966 14462 leader_election.cc:288] T 00000000000000000000000000000000 P 02a95480a8ca4c6cb131b248b6f5de92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:49:38.906062 14467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02a95480a8ca4c6cb131b248b6f5de92 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:49:38.906276 14467 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02a95480a8ca4c6cb131b248b6f5de92 [term 1 LEADER]: Becoming Leader. State: Replica: 02a95480a8ca4c6cb131b248b6f5de92, State: Running, Role: LEADER
I20260621 07:49:38.906556 14467 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02a95480a8ca4c6cb131b248b6f5de92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02a95480a8ca4c6cb131b248b6f5de92" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45645 } }
I20260621 07:49:38.907006 14462 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02a95480a8ca4c6cb131b248b6f5de92 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:49:38.908006 14469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02a95480a8ca4c6cb131b248b6f5de92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02a95480a8ca4c6cb131b248b6f5de92. Latest consensus state: current_term: 1 leader_uuid: "02a95480a8ca4c6cb131b248b6f5de92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02a95480a8ca4c6cb131b248b6f5de92" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45645 } } }
I20260621 07:49:38.908126 14469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02a95480a8ca4c6cb131b248b6f5de92 [sys.catalog]: This master's current role is: LEADER
I20260621 07:49:38.908412 14468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02a95480a8ca4c6cb131b248b6f5de92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02a95480a8ca4c6cb131b248b6f5de92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02a95480a8ca4c6cb131b248b6f5de92" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45645 } } }
I20260621 07:49:38.908497 14476 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:49:38.908519 14468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02a95480a8ca4c6cb131b248b6f5de92 [sys.catalog]: This master's current role is: LEADER
I20260621 07:49:38.910663 14476 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:49:38.915129 14476 catalog_manager.cc:1347] Generated new cluster ID: 1d3a35bb002149fcbbfb48d6e7ffc561
I20260621 07:49:38.915180 14476 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:49:38.939226 14476 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:49:38.939944 14476 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:49:38.944649 14476 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02a95480a8ca4c6cb131b248b6f5de92: Generated new TSK 0
I20260621 07:49:38.945138 14476 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:49:38.959570 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 14394
2026-06-21T07:49:38Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1315 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:49:39 dist-test-slave-lzx3 krb5kdc[14489](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:49:39 dist-test-slave-lzx3 krb5kdc[14489](info): set up 2 sockets
Jun 21 07:49:39 dist-test-slave-lzx3 krb5kdc[14489](info): commencing operation
krb5kdc: starting...
W20260621 07:49:39.928416 771 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/test-user.keytab.
Jun 21 07:49:39 dist-test-slave-lzx3 krb5kdc[14489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:49:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:49:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.192.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
I20260621 07:49:40.029565 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:42677
--webserver_interface=127.0.192.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:36019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:42677
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:49:40.125765 14502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:49:40.125948 14502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:49:40.125979 14502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:49:40.129611 14502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:49:40.129652 14502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:49:40.129670 14502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:49:40.129686 14502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:49:40.129700 14502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:49:40.134105 14502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:36019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:42677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:42677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:49:40.134814 14502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:49:40.135546 14502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:49:40.142729 14507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:49:40.142710 14508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:49:40.142803 14502 server_base.cc:1034] running on GCE node
W20260621 07:49:40.142741 14510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:49:40.143520 14502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:49:40.144418 14502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:49:40.145562 14502 hybrid_clock.cc:648] HybridClock initialized: now 1782028180145533 us; error 43 us; skew 500 ppm
Jun 21 07:49:40 dist-test-slave-lzx3 krb5kdc[14489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:49:40.148125 14502 init.cc:377] Logged in from keytab as kudu/127.0.192.254@KRBTEST.COM (short username kudu)
I20260621 07:49:40.148913 14502 webserver.cc:460] Webserver started at http://127.0.192.254:41761/ using document root <none> and password file <none>
I20260621 07:49:40.149374 14502 fs_manager.cc:359] Metadata directory not provided
I20260621 07:49:40.149427 14502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:49:40.149636 14502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:49:40.151228 14502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "814aaaa5c60c4ca6a09cde0649decd1a"
format_stamp: "Formatted at 2026-06-21 07:49:40 on dist-test-slave-lzx3"
server_key: "514c1a28bb9e41ac7f516823c4ee3437"
server_key_iv: "9eddee813aeb52d38b7a1e12ca8afa73"
server_key_version: "encryptionkey@0"
I20260621 07:49:40.151698 14502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "814aaaa5c60c4ca6a09cde0649decd1a"
format_stamp: "Formatted at 2026-06-21 07:49:40 on dist-test-slave-lzx3"
server_key: "514c1a28bb9e41ac7f516823c4ee3437"
server_key_iv: "9eddee813aeb52d38b7a1e12ca8afa73"
server_key_version: "encryptionkey@0"
I20260621 07:49:40.154875 14502 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 07:49:40.156905 14517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:40.157900 14502 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:49:40.158007 14502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "814aaaa5c60c4ca6a09cde0649decd1a"
format_stamp: "Formatted at 2026-06-21 07:49:40 on dist-test-slave-lzx3"
server_key: "514c1a28bb9e41ac7f516823c4ee3437"
server_key_iv: "9eddee813aeb52d38b7a1e12ca8afa73"
server_key_version: "encryptionkey@0"
I20260621 07:49:40.158094 14502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:49:40.167244 14502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:49:40.169763 14502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:49:40.169931 14502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:49:40.175374 14502 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:42677
I20260621 07:49:40.175609 14569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:42677 every 8 connection(s)
I20260621 07:49:40.176194 14502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:49:40.178483 14570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:49:40.182727 14570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a: Bootstrap starting.
I20260621 07:49:40.184422 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 14502
I20260621 07:49:40.184531 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:49:40.184785 771 external_mini_cluster.cc:1428] Setting key 7b66300291b46b86557b4209eec41e1d
I20260621 07:49:40.184953 14570 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a: Neither blocks nor log segments found. Creating new log.
I20260621 07:49:40.185688 14570 log.cc:824] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a: Log is configured to *not* fsync() on all Append() calls
I20260621 07:49:40.187332 14570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a: No bootstrap required, opened a new log
I20260621 07:49:40.190727 14570 raft_consensus.cc:348] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814aaaa5c60c4ca6a09cde0649decd1a" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 42677 } }
I20260621 07:49:40.190958 14570 raft_consensus.cc:374] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:49:40.191008 14570 raft_consensus.cc:729] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 814aaaa5c60c4ca6a09cde0649decd1a, State: Initialized, Role: FOLLOWER
I20260621 07:49:40.191421 14570 consensus_queue.cc:260] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814aaaa5c60c4ca6a09cde0649decd1a" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 42677 } }
I20260621 07:49:40.191546 14570 raft_consensus.cc:388] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:49:40.191614 14570 raft_consensus.cc:482] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:49:40.191715 14570 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [term 0 FOLLOWER]: Advancing to term 1
Jun 21 07:49:40 dist-test-slave-lzx3 krb5kdc[14489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:49:40.192447 14570 raft_consensus.cc:504] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814aaaa5c60c4ca6a09cde0649decd1a" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 42677 } }
I20260621 07:49:40.192770 14570 leader_election.cc:302] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 814aaaa5c60c4ca6a09cde0649decd1a; no voters:
I20260621 07:49:40.193010 14570 leader_election.cc:288] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:49:40.193249 14575 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:49:40.193619 14575 raft_consensus.cc:686] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [term 1 LEADER]: Becoming Leader. State: Replica: 814aaaa5c60c4ca6a09cde0649decd1a, State: Running, Role: LEADER
I20260621 07:49:40.193981 14575 consensus_queue.cc:237] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814aaaa5c60c4ca6a09cde0649decd1a" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 42677 } }
I20260621 07:49:40.194090 14570 sys_catalog.cc:564] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:49:40.195214 14577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 814aaaa5c60c4ca6a09cde0649decd1a. Latest consensus state: current_term: 1 leader_uuid: "814aaaa5c60c4ca6a09cde0649decd1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814aaaa5c60c4ca6a09cde0649decd1a" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 42677 } } }
I20260621 07:49:40.195322 14577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [sys.catalog]: This master's current role is: LEADER
I20260621 07:49:40.195345 14576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "814aaaa5c60c4ca6a09cde0649decd1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814aaaa5c60c4ca6a09cde0649decd1a" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 42677 } } }
I20260621 07:49:40.195487 14576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [sys.catalog]: This master's current role is: LEADER
I20260621 07:49:40.197811 14573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:40.185678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41548 (local address 127.0.192.254:42677)
0621 07:49:40.186010 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:40.186023 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:40.186050 (+ 27us) server_negotiation.cc:407] Connection header received
0621 07:49:40.186589 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:40.186603 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:40.186877 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:40.187118 (+ 241us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:49:40.187919 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:40.188770 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:40.189848 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:40.190207 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:40.190305 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:40.192652 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:49:40.192657 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:49:40.192667 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:49:40.192689 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:49:40.194548 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:40.194986 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:40.194990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:40.194993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:40.195080 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:40.195375 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:40.195378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:40.195380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:40.195812 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:49:40.195988 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:40.196705 (+ 717us) server_negotiation.cc:299] Negotiation successful
0621 07:49:40.196970 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":122,"threads_started":1}
W20260621 07:49:40.198282 14590 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:49:40.198352 14590 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:49:40.198448 14585 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:49:40.199151 14585 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:49:40.203487 14585 catalog_manager.cc:1347] Generated new cluster ID: 138f75245e30495293a83cb39ad5ff64
I20260621 07:49:40.203538 14585 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:49:40.229193 14585 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:49:40.229935 14585 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:49:40.234861 14585 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a: Generated new TSK 0
I20260621 07:49:40.235333 14585 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.192.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab.
Entry for principal kudu/127.0.192.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab.
I20260621 07:49:40.260542 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.193:0
--local_ip_for_outbound_sockets=127.0.192.193
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:42677
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:36019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:49:40.356909 14596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:49:40.357103 14596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:49:40.357170 14596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:49:40.360509 14596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:49:40.360599 14596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:49:40.360688 14596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.193
I20260621 07:49:40.365126 14596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:36019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:42677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.193
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:49:40.365806 14596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:49:40.366571 14596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 07:49:40.374651 14596 server_base.cc:1034] running on GCE node
W20260621 07:49:40.374639 14601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:49:40.374799 14602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:49:40.374684 14604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:49:40.375177 14596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:49:40.375660 14596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:49:40.376813 14596 hybrid_clock.cc:648] HybridClock initialized: now 1782028180376804 us; error 23 us; skew 500 ppm
Jun 21 07:49:40 dist-test-slave-lzx3 krb5kdc[14489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:49:40.379312 14596 init.cc:377] Logged in from keytab as kudu/127.0.192.193@KRBTEST.COM (short username kudu)
I20260621 07:49:40.380120 14596 webserver.cc:460] Webserver started at http://127.0.192.193:46859/ using document root <none> and password file <none>
I20260621 07:49:40.380673 14596 fs_manager.cc:359] Metadata directory not provided
I20260621 07:49:40.380744 14596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:49:40.380934 14596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:49:40.382597 14596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/data/instance:
uuid: "74936c7336d24fd3b3070e3ff5b2105b"
format_stamp: "Formatted at 2026-06-21 07:49:40 on dist-test-slave-lzx3"
server_key: "619b4bc1ac893ba2ce926b0f86126a56"
server_key_iv: "b462ad764d7f3735f8c06fda5a8ac4e0"
server_key_version: "encryptionkey@0"
I20260621 07:49:40.383119 14596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/wal/instance:
uuid: "74936c7336d24fd3b3070e3ff5b2105b"
format_stamp: "Formatted at 2026-06-21 07:49:40 on dist-test-slave-lzx3"
server_key: "619b4bc1ac893ba2ce926b0f86126a56"
server_key_iv: "b462ad764d7f3735f8c06fda5a8ac4e0"
server_key_version: "encryptionkey@0"
I20260621 07:49:40.386312 14596 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 07:49:40.388437 14611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:40.389541 14596 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:49:40.389674 14596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/wal
uuid: "74936c7336d24fd3b3070e3ff5b2105b"
format_stamp: "Formatted at 2026-06-21 07:49:40 on dist-test-slave-lzx3"
server_key: "619b4bc1ac893ba2ce926b0f86126a56"
server_key_iv: "b462ad764d7f3735f8c06fda5a8ac4e0"
server_key_version: "encryptionkey@0"
I20260621 07:49:40.389791 14596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:49:40.414129 14596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:49:40.416456 14596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:49:40.416671 14596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:49:40.417160 14596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:49:40.417857 14596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:49:40.417920 14596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:40.417980 14596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:49:40.418022 14596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:40.426040 14596 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.193:36707
I20260621 07:49:40.426060 14724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.193:36707 every 8 connection(s)
I20260621 07:49:40.426817 14596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
Jun 21 07:49:40 dist-test-slave-lzx3 krb5kdc[14489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:49:40.435626 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 14596
I20260621 07:49:40.435741 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/wal/instance
I20260621 07:49:40.436029 771 external_mini_cluster.cc:1428] Setting key 4bb161eb86a31188e4b84125ac38407c
I20260621 07:49:40.437438 14573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:40.428275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:40139 (local address 127.0.192.254:42677)
0621 07:49:40.428383 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:40.428387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:40.429194 (+ 807us) server_negotiation.cc:407] Connection header received
0621 07:49:40.429834 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:40.429836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:40.429874 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:40.429935 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:49:40.430815 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:40.431283 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:40.432147 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:40.432346 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:40.432412 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:40.434383 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:49:40.434386 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:49:40.434387 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:49:40.434407 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:49:40.435395 (+ 988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:40.435902 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:40.435904 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:40.435905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:40.435970 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:40.436342 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:40.436344 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:40.436345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:40.436592 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:49:40.436684 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:40.437154 (+ 470us) server_negotiation.cc:299] Negotiation successful
0621 07:49:40.437266 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260621 07:49:40.438200 14727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:40.428511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.193:40139)
0621 07:49:40.429073 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:40.429104 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0621 07:49:40.429691 (+ 587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:49:40.430055 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:49:40.430060 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:49:40.430340 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:49:40.430707 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:40.430717 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:40.431371 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:40.431373 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:40.432059 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:40.432066 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:40.432452 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:40.432455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:40.432613 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:40.433077 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:49:40.433090 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:49:40.434260 (+ 1170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:49:40.435491 (+ 1231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:40.435497 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:40.435509 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:40.435814 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:40.436051 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:40.436054 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:40.436056 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:40.436243 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:40.436701 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:49:40.436714 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:49:40.436934 (+ 220us) client_negotiation.cc:769] Sending connection context
0621 07:49:40.437079 (+ 145us) client_negotiation.cc:240] Negotiation successful
0621 07:49:40.437389 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":98,"threads_started":1}
I20260621 07:49:40.439354 14725 heartbeater.cc:347] Connected to a master server at 127.0.192.254:42677
I20260621 07:49:40.439543 14725 heartbeater.cc:464] Registering TS with master...
I20260621 07:49:40.439913 14725 heartbeater.cc:511] Master 127.0.192.254:42677 requested a full tablet report, sending...
I20260621 07:49:40.441154 14534 ts_manager.cc:194] Registered new tserver with Master: 74936c7336d24fd3b3070e3ff5b2105b (127.0.192.193:36707)
I20260621 07:49:40.442127 14534 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.193@KRBTEST.COM'} at 127.0.192.193:40139
WARNING: no policy specified for kudu/127.0.192.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab.
Entry for principal kudu/127.0.192.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab.
I20260621 07:49:40.460963 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.194:0
--local_ip_for_outbound_sockets=127.0.192.194
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:42677
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:36019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:49:40.554961 14730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:49:40.555162 14730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:49:40.555197 14730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:49:40.558647 14730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:49:40.558684 14730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:49:40.558729 14730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.194
I20260621 07:49:40.563102 14730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:36019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:42677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.194
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:49:40.563812 14730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:49:40.564527 14730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:49:40.572233 14735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:49:40.572328 14730 server_base.cc:1034] running on GCE node
W20260621 07:49:40.572260 14736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:49:40.572573 14738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:49:40.572926 14730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:49:40.573470 14730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:49:40.574667 14730 hybrid_clock.cc:648] HybridClock initialized: now 1782028180574646 us; error 37 us; skew 500 ppm
Jun 21 07:49:40 dist-test-slave-lzx3 krb5kdc[14489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:49:40.577239 14730 init.cc:377] Logged in from keytab as kudu/127.0.192.194@KRBTEST.COM (short username kudu)
I20260621 07:49:40.577977 14730 webserver.cc:460] Webserver started at http://127.0.192.194:33447/ using document root <none> and password file <none>
I20260621 07:49:40.578440 14730 fs_manager.cc:359] Metadata directory not provided
I20260621 07:49:40.578533 14730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:49:40.578735 14730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:49:40.580472 14730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/data/instance:
uuid: "a0f1558e9a154c5c9f2c171095b2d466"
format_stamp: "Formatted at 2026-06-21 07:49:40 on dist-test-slave-lzx3"
server_key: "d9b79482b0666d353e0c86f31c4c0f49"
server_key_iv: "d4705c943e09bad2d1a0d6837cb9abd2"
server_key_version: "encryptionkey@0"
I20260621 07:49:40.581061 14730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/wal/instance:
uuid: "a0f1558e9a154c5c9f2c171095b2d466"
format_stamp: "Formatted at 2026-06-21 07:49:40 on dist-test-slave-lzx3"
server_key: "d9b79482b0666d353e0c86f31c4c0f49"
server_key_iv: "d4705c943e09bad2d1a0d6837cb9abd2"
server_key_version: "encryptionkey@0"
I20260621 07:49:40.584273 14730 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:49:40.586392 14745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:40.587481 14730 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:49:40.587597 14730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/wal
uuid: "a0f1558e9a154c5c9f2c171095b2d466"
format_stamp: "Formatted at 2026-06-21 07:49:40 on dist-test-slave-lzx3"
server_key: "d9b79482b0666d353e0c86f31c4c0f49"
server_key_iv: "d4705c943e09bad2d1a0d6837cb9abd2"
server_key_version: "encryptionkey@0"
I20260621 07:49:40.587721 14730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:49:40.602082 14730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:49:40.605237 14730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:49:40.605403 14730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:49:40.605875 14730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:49:40.606606 14730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:49:40.606652 14730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:40.606688 14730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:49:40.606704 14730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:40.615445 14730 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.194:36799
I20260621 07:49:40.615618 14858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.194:36799 every 8 connection(s)
I20260621 07:49:40.616289 14730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
Jun 21 07:49:40 dist-test-slave-lzx3 krb5kdc[14489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:49:40.625660 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 14730
I20260621 07:49:40.626060 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/wal/instance
I20260621 07:49:40.626398 771 external_mini_cluster.cc:1428] Setting key f39dbea89a4c471f1426acd936662563
I20260621 07:49:40.629330 14573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:40.617914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:34207 (local address 127.0.192.254:42677)
0621 07:49:40.618061 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:40.618065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:40.619046 (+ 981us) server_negotiation.cc:407] Connection header received
0621 07:49:40.619940 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:40.619943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:40.620009 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:40.620108 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:49:40.621287 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:40.621830 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:40.622943 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:40.623172 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:40.623280 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:40.625501 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:49:40.625507 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:49:40.625510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:49:40.625539 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:49:40.627179 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:40.627706 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:40.627709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:40.627711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:40.627795 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:40.628139 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:40.628143 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:40.628144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:40.628362 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:49:40.628462 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:40.628950 (+ 488us) server_negotiation.cc:299] Negotiation successful
0621 07:49:40.629081 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 07:49:40.629858 14861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:40.618257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.194:34207)
0621 07:49:40.618897 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:40.618934 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 07:49:40.619744 (+ 810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:49:40.620275 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:49:40.620282 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:49:40.620712 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:49:40.621152 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:40.621164 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:40.621951 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:40.621956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:40.622819 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:40.622827 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:40.623310 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:40.623314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:40.623479 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:40.623940 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:49:40.623954 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:49:40.625329 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:49:40.627285 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:40.627291 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:40.627316 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:40.627613 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:40.627886 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:40.627889 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:40.627891 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:40.628038 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:40.628467 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:49:40.628471 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:49:40.628710 (+ 239us) client_negotiation.cc:769] Sending connection context
0621 07:49:40.628856 (+ 146us) client_negotiation.cc:240] Negotiation successful
0621 07:49:40.629109 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":106,"threads_started":1}
I20260621 07:49:40.630892 14859 heartbeater.cc:347] Connected to a master server at 127.0.192.254:42677
I20260621 07:49:40.631106 14859 heartbeater.cc:464] Registering TS with master...
I20260621 07:49:40.631522 14859 heartbeater.cc:511] Master 127.0.192.254:42677 requested a full tablet report, sending...
I20260621 07:49:40.632522 14534 ts_manager.cc:194] Registered new tserver with Master: a0f1558e9a154c5c9f2c171095b2d466 (127.0.192.194:36799)
I20260621 07:49:40.633340 14534 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.194@KRBTEST.COM'} at 127.0.192.194:34207
WARNING: no policy specified for kudu/127.0.192.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab.
Entry for principal kudu/127.0.192.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab.
I20260621 07:49:40.651170 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.195:0
--local_ip_for_outbound_sockets=127.0.192.195
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:42677
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:36019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:49:40.750018 14864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:49:40.750204 14864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:49:40.750237 14864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:49:40.753567 14864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:49:40.753610 14864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:49:40.753659 14864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.195
I20260621 07:49:40.758279 14864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:36019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:42677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.195
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:49:40.759022 14864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:49:40.759797 14864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:49:40.768018 14869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:49:40.768035 14870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:49:40.768141 14864 server_base.cc:1034] running on GCE node
W20260621 07:49:40.768009 14872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:49:40.768616 14864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:49:40.769156 14864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:49:40.770325 14864 hybrid_clock.cc:648] HybridClock initialized: now 1782028180770304 us; error 30 us; skew 500 ppm
Jun 21 07:49:40 dist-test-slave-lzx3 krb5kdc[14489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:49:40.772795 14864 init.cc:377] Logged in from keytab as kudu/127.0.192.195@KRBTEST.COM (short username kudu)
I20260621 07:49:40.773432 14864 webserver.cc:460] Webserver started at http://127.0.192.195:38937/ using document root <none> and password file <none>
I20260621 07:49:40.773826 14864 fs_manager.cc:359] Metadata directory not provided
I20260621 07:49:40.773873 14864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:49:40.774008 14864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:49:40.775641 14864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/data/instance:
uuid: "ee318ad5e75847f49a88290b7d43fcde"
format_stamp: "Formatted at 2026-06-21 07:49:40 on dist-test-slave-lzx3"
server_key: "7f7ea3f47e3b338096e6055e96534c32"
server_key_iv: "e68ad8c496595a39d94c6e5694352ec9"
server_key_version: "encryptionkey@0"
I20260621 07:49:40.776124 14864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/wal/instance:
uuid: "ee318ad5e75847f49a88290b7d43fcde"
format_stamp: "Formatted at 2026-06-21 07:49:40 on dist-test-slave-lzx3"
server_key: "7f7ea3f47e3b338096e6055e96534c32"
server_key_iv: "e68ad8c496595a39d94c6e5694352ec9"
server_key_version: "encryptionkey@0"
I20260621 07:49:40.779398 14864 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 07:49:40.781430 14879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:40.782604 14864 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:49:40.782749 14864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/wal
uuid: "ee318ad5e75847f49a88290b7d43fcde"
format_stamp: "Formatted at 2026-06-21 07:49:40 on dist-test-slave-lzx3"
server_key: "7f7ea3f47e3b338096e6055e96534c32"
server_key_iv: "e68ad8c496595a39d94c6e5694352ec9"
server_key_version: "encryptionkey@0"
I20260621 07:49:40.782855 14864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:49:40.813894 14864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:49:40.816406 14864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:49:40.816637 14864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:49:40.817184 14864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:49:40.817981 14864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:49:40.818029 14864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:40.818096 14864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:49:40.818133 14864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:40.827091 14864 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.195:43655
I20260621 07:49:40.827117 14992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.195:43655 every 8 connection(s)
I20260621 07:49:40.827916 14864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
I20260621 07:49:40.836040 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 14864
I20260621 07:49:40.836135 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/wal/instance
I20260621 07:49:40.836409 771 external_mini_cluster.cc:1428] Setting key 555489de541119aabccc2f74bc796618
Jun 21 07:49:40 dist-test-slave-lzx3 krb5kdc[14489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:49:40.840736 14573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:40.829649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:59751 (local address 127.0.192.254:42677)
0621 07:49:40.829779 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:40.829783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:40.830721 (+ 938us) server_negotiation.cc:407] Connection header received
0621 07:49:40.831544 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:40.831547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:40.831604 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:40.831676 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:49:40.832751 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:40.833257 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:40.834352 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:40.834574 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:40.834693 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:40.837160 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:49:40.837164 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:49:40.837167 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:49:40.837201 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:49:40.838767 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:40.839285 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:40.839290 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:40.839291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:40.839383 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:40.839724 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:40.839728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:40.839730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:40.839904 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:49:40.840006 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:40.840444 (+ 438us) server_negotiation.cc:299] Negotiation successful
0621 07:49:40.840571 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 07:49:40.841393 14995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:40.830000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.195:59751)
0621 07:49:40.830594 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:40.830628 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0621 07:49:40.831352 (+ 724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:49:40.831830 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:49:40.831836 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:49:40.832150 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:49:40.832600 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:40.832626 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:40.833393 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:40.833399 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:40.834229 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:40.834238 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:40.834730 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:40.834734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:40.834895 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:40.835348 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:49:40.835362 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:49:40.836994 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:49:40.838903 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:40.838908 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:40.838921 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:40.839167 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:40.839483 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:40.839486 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:40.839488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:40.839625 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:40.840015 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:49:40.840023 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:49:40.840224 (+ 201us) client_negotiation.cc:769] Sending connection context
0621 07:49:40.840401 (+ 177us) client_negotiation.cc:240] Negotiation successful
0621 07:49:40.840697 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":120,"threads_started":1}
I20260621 07:49:40.842383 14993 heartbeater.cc:347] Connected to a master server at 127.0.192.254:42677
I20260621 07:49:40.842582 14993 heartbeater.cc:464] Registering TS with master...
I20260621 07:49:40.843032 14993 heartbeater.cc:511] Master 127.0.192.254:42677 requested a full tablet report, sending...
I20260621 07:49:40.843952 14534 ts_manager.cc:194] Registered new tserver with Master: ee318ad5e75847f49a88290b7d43fcde (127.0.192.195:43655)
I20260621 07:49:40.844460 14534 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.195@KRBTEST.COM'} at 127.0.192.195:59751
I20260621 07:49:40.849905 771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:49:40.858037 14573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:40.851588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41564 (local address 127.0.192.254:42677)
0621 07:49:40.851718 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:40.851724 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:40.851834 (+ 110us) server_negotiation.cc:407] Connection header received
0621 07:49:40.851927 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:40.851929 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:40.851962 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:40.852024 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:49:40.852630 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:40.853073 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:40.853812 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:40.853999 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:40.854067 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:40.855135 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:49:40.855137 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:49:40.855139 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:49:40.855158 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:49:40.856539 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:40.856919 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:40.856922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:40.856923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:40.856989 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:40.857244 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:40.857247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:40.857248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:40.857458 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:49:40.857579 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:40.857732 (+ 153us) server_negotiation.cc:299] Negotiation successful
0621 07:49:40.857809 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 07:49:40.860412 14534 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:49:40.863248 14534 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:49:40.875732 15004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:40.870642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:36707 (local address 127.0.0.1:51806)
0621 07:49:40.871197 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:40.871244 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0621 07:49:40.871359 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:49:40.872186 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:49:40.872191 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:49:40.872209 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:49:40.872314 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:40.872323 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:40.873447 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:40.873452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:40.874523 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:40.874531 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:40.875297 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:40.875301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:40.875396 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:40.875427 (+ 31us) client_negotiation.cc:769] Sending connection context
0621 07:49:40.875525 (+ 98us) client_negotiation.cc:240] Negotiation successful
0621 07:49:40.875621 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":77,"threads_started":1}
I20260621 07:49:40.875744 15005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:40.870830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51806 (local address 127.0.192.193:36707)
0621 07:49:40.871735 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:40.871748 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:40.871769 (+ 21us) server_negotiation.cc:407] Connection header received
0621 07:49:40.871841 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:40.871846 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:40.872058 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:40.872198 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:49:40.872437 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:40.873318 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:40.874642 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:40.875186 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:40.875297 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:40.875429 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:40.875551 (+ 122us) server_negotiation.cc:299] Negotiation successful
0621 07:49:40.875628 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":806,"thread_start_us":133,"threads_started":1}
I20260621 07:49:40.876590 15007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:40.871300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.195:43655 (local address 127.0.0.1:58034)
0621 07:49:40.871952 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:40.871965 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:49:40.872042 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:49:40.872642 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:49:40.872645 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:49:40.872665 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:49:40.872745 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:40.872751 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:40.874032 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:40.874035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:40.875201 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:40.875211 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:40.876189 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:40.876193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:40.876281 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:40.876302 (+ 21us) client_negotiation.cc:769] Sending connection context
0621 07:49:40.876392 (+ 90us) client_negotiation.cc:240] Negotiation successful
0621 07:49:40.876452 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":91,"threads_started":1}
I20260621 07:49:40.876688 15009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:40.871889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58034 (local address 127.0.192.195:43655)
0621 07:49:40.872171 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:40.872183 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:40.872206 (+ 23us) server_negotiation.cc:407] Connection header received
0621 07:49:40.872271 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:40.872276 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:40.872449 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:40.872670 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:49:40.872837 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:40.873869 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:40.875349 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:40.876056 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:40.876174 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:40.876305 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:40.876416 (+ 111us) server_negotiation.cc:299] Negotiation successful
0621 07:49:40.876511 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":125,"threads_started":1}
I20260621 07:49:40.876979 15006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:40.871117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:36799 (local address 127.0.0.1:45044)
0621 07:49:40.871589 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:40.871603 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:49:40.871699 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:49:40.872747 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:49:40.872751 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:49:40.872769 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:49:40.872890 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:40.872898 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:40.874406 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:40.874410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:40.875499 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:40.875508 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:40.876616 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:40.876620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:40.876719 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:40.876747 (+ 28us) client_negotiation.cc:769] Sending connection context
0621 07:49:40.876810 (+ 63us) client_negotiation.cc:240] Negotiation successful
0621 07:49:40.876891 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":3,"thread_start_us":84,"threads_started":1}
I20260621 07:49:40.877612 15008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:40.871527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45044 (local address 127.0.192.194:36799)
0621 07:49:40.872292 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:40.872304 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:40.872324 (+ 20us) server_negotiation.cc:407] Connection header received
0621 07:49:40.872393 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:40.872397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:40.872574 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:40.873057 (+ 483us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:49:40.873213 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:40.874257 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:40.875640 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:40.876455 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:40.877176 (+ 721us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:40.877332 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:40.877380 (+ 48us) server_negotiation.cc:299] Negotiation successful
0621 07:49:40.877463 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":143,"threads_started":1}
I20260621 07:49:40.878798 14659 tablet_service.cc:1467] Processing CreateTablet for tablet b1fdcb1d8f474d9b9c911ad7015f9387 (DEFAULT_TABLE table=test-table [id=9093818f183247cc8c6da0b1195984ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:49:40.879686 14659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1fdcb1d8f474d9b9c911ad7015f9387. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:49:40.879807 14793 tablet_service.cc:1467] Processing CreateTablet for tablet b1fdcb1d8f474d9b9c911ad7015f9387 (DEFAULT_TABLE table=test-table [id=9093818f183247cc8c6da0b1195984ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:49:40.880695 14793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1fdcb1d8f474d9b9c911ad7015f9387. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:49:40.885452 15010 tablet_bootstrap.cc:492] T b1fdcb1d8f474d9b9c911ad7015f9387 P 74936c7336d24fd3b3070e3ff5b2105b: Bootstrap starting.
I20260621 07:49:40.886588 15011 tablet_bootstrap.cc:492] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466: Bootstrap starting.
I20260621 07:49:40.886307 14927 tablet_service.cc:1467] Processing CreateTablet for tablet b1fdcb1d8f474d9b9c911ad7015f9387 (DEFAULT_TABLE table=test-table [id=9093818f183247cc8c6da0b1195984ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:49:40.887223 14927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1fdcb1d8f474d9b9c911ad7015f9387. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:49:40.888147 15010 tablet_bootstrap.cc:654] T b1fdcb1d8f474d9b9c911ad7015f9387 P 74936c7336d24fd3b3070e3ff5b2105b: Neither blocks nor log segments found. Creating new log.
I20260621 07:49:40.888862 15011 tablet_bootstrap.cc:654] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466: Neither blocks nor log segments found. Creating new log.
I20260621 07:49:40.889209 15010 log.cc:824] T b1fdcb1d8f474d9b9c911ad7015f9387 P 74936c7336d24fd3b3070e3ff5b2105b: Log is configured to *not* fsync() on all Append() calls
I20260621 07:49:40.889585 15011 log.cc:824] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466: Log is configured to *not* fsync() on all Append() calls
I20260621 07:49:40.890964 15010 tablet_bootstrap.cc:492] T b1fdcb1d8f474d9b9c911ad7015f9387 P 74936c7336d24fd3b3070e3ff5b2105b: No bootstrap required, opened a new log
I20260621 07:49:40.891170 15010 ts_tablet_manager.cc:1397] T b1fdcb1d8f474d9b9c911ad7015f9387 P 74936c7336d24fd3b3070e3ff5b2105b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260621 07:49:40.891315 15011 tablet_bootstrap.cc:492] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466: No bootstrap required, opened a new log
I20260621 07:49:40.891470 15011 ts_tablet_manager.cc:1397] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260621 07:49:40.893136 15014 tablet_bootstrap.cc:492] T b1fdcb1d8f474d9b9c911ad7015f9387 P ee318ad5e75847f49a88290b7d43fcde: Bootstrap starting.
I20260621 07:49:40.894868 15011 raft_consensus.cc:348] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74936c7336d24fd3b3070e3ff5b2105b" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 36707 } } peers { permanent_uuid: "ee318ad5e75847f49a88290b7d43fcde" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43655 } } peers { permanent_uuid: "a0f1558e9a154c5c9f2c171095b2d466" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 36799 } }
I20260621 07:49:40.895067 15011 raft_consensus.cc:374] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:49:40.894933 15010 raft_consensus.cc:348] T b1fdcb1d8f474d9b9c911ad7015f9387 P 74936c7336d24fd3b3070e3ff5b2105b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74936c7336d24fd3b3070e3ff5b2105b" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 36707 } } peers { permanent_uuid: "ee318ad5e75847f49a88290b7d43fcde" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43655 } } peers { permanent_uuid: "a0f1558e9a154c5c9f2c171095b2d466" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 36799 } }
I20260621 07:49:40.895114 15011 raft_consensus.cc:729] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0f1558e9a154c5c9f2c171095b2d466, State: Initialized, Role: FOLLOWER
I20260621 07:49:40.895152 15010 raft_consensus.cc:374] T b1fdcb1d8f474d9b9c911ad7015f9387 P 74936c7336d24fd3b3070e3ff5b2105b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:49:40.895237 15010 raft_consensus.cc:729] T b1fdcb1d8f474d9b9c911ad7015f9387 P 74936c7336d24fd3b3070e3ff5b2105b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74936c7336d24fd3b3070e3ff5b2105b, State: Initialized, Role: FOLLOWER
I20260621 07:49:40.895594 15014 tablet_bootstrap.cc:654] T b1fdcb1d8f474d9b9c911ad7015f9387 P ee318ad5e75847f49a88290b7d43fcde: Neither blocks nor log segments found. Creating new log.
I20260621 07:49:40.895555 15011 consensus_queue.cc:260] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74936c7336d24fd3b3070e3ff5b2105b" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 36707 } } peers { permanent_uuid: "ee318ad5e75847f49a88290b7d43fcde" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43655 } } peers { permanent_uuid: "a0f1558e9a154c5c9f2c171095b2d466" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 36799 } }
I20260621 07:49:40.895697 15010 consensus_queue.cc:260] T b1fdcb1d8f474d9b9c911ad7015f9387 P 74936c7336d24fd3b3070e3ff5b2105b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74936c7336d24fd3b3070e3ff5b2105b" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 36707 } } peers { permanent_uuid: "ee318ad5e75847f49a88290b7d43fcde" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43655 } } peers { permanent_uuid: "a0f1558e9a154c5c9f2c171095b2d466" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 36799 } }
I20260621 07:49:40.896230 14859 heartbeater.cc:503] Master 127.0.192.254:42677 was elected leader, sending a full tablet report...
I20260621 07:49:40.896413 15014 log.cc:824] T b1fdcb1d8f474d9b9c911ad7015f9387 P ee318ad5e75847f49a88290b7d43fcde: Log is configured to *not* fsync() on all Append() calls
I20260621 07:49:40.896512 15011 ts_tablet_manager.cc:1428] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260621 07:49:40.897684 15010 ts_tablet_manager.cc:1428] T b1fdcb1d8f474d9b9c911ad7015f9387 P 74936c7336d24fd3b3070e3ff5b2105b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260621 07:49:40.898007 14725 heartbeater.cc:503] Master 127.0.192.254:42677 was elected leader, sending a full tablet report...
I20260621 07:49:40.898209 15014 tablet_bootstrap.cc:492] T b1fdcb1d8f474d9b9c911ad7015f9387 P ee318ad5e75847f49a88290b7d43fcde: No bootstrap required, opened a new log
I20260621 07:49:40.898341 15014 ts_tablet_manager.cc:1397] T b1fdcb1d8f474d9b9c911ad7015f9387 P ee318ad5e75847f49a88290b7d43fcde: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260621 07:49:40.901738 15014 raft_consensus.cc:348] T b1fdcb1d8f474d9b9c911ad7015f9387 P ee318ad5e75847f49a88290b7d43fcde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74936c7336d24fd3b3070e3ff5b2105b" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 36707 } } peers { permanent_uuid: "ee318ad5e75847f49a88290b7d43fcde" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43655 } } peers { permanent_uuid: "a0f1558e9a154c5c9f2c171095b2d466" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 36799 } }
I20260621 07:49:40.901933 15014 raft_consensus.cc:374] T b1fdcb1d8f474d9b9c911ad7015f9387 P ee318ad5e75847f49a88290b7d43fcde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:49:40.902007 15014 raft_consensus.cc:729] T b1fdcb1d8f474d9b9c911ad7015f9387 P ee318ad5e75847f49a88290b7d43fcde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee318ad5e75847f49a88290b7d43fcde, State: Initialized, Role: FOLLOWER
I20260621 07:49:40.902372 15014 consensus_queue.cc:260] T b1fdcb1d8f474d9b9c911ad7015f9387 P ee318ad5e75847f49a88290b7d43fcde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74936c7336d24fd3b3070e3ff5b2105b" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 36707 } } peers { permanent_uuid: "ee318ad5e75847f49a88290b7d43fcde" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43655 } } peers { permanent_uuid: "a0f1558e9a154c5c9f2c171095b2d466" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 36799 } }
I20260621 07:49:40.902948 14993 heartbeater.cc:503] Master 127.0.192.254:42677 was elected leader, sending a full tablet report...
I20260621 07:49:40.903167 15014 ts_tablet_manager.cc:1428] T b1fdcb1d8f474d9b9c911ad7015f9387 P ee318ad5e75847f49a88290b7d43fcde: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260621 07:49:40.927815 14726 tablet.cc:2369] T b1fdcb1d8f474d9b9c911ad7015f9387 P 74936c7336d24fd3b3070e3ff5b2105b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 07:49:41.078918 14994 tablet.cc:2369] T b1fdcb1d8f474d9b9c911ad7015f9387 P ee318ad5e75847f49a88290b7d43fcde: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 07:49:41.117389 14860 tablet.cc:2369] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:49:41.222595 15015 raft_consensus.cc:482] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:49:41.222786 15015 raft_consensus.cc:504] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74936c7336d24fd3b3070e3ff5b2105b" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 36707 } } peers { permanent_uuid: "ee318ad5e75847f49a88290b7d43fcde" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43655 } } peers { permanent_uuid: "a0f1558e9a154c5c9f2c171095b2d466" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 36799 } }
I20260621 07:49:41.223819 15015 leader_election.cc:288] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74936c7336d24fd3b3070e3ff5b2105b (127.0.192.193:36707), ee318ad5e75847f49a88290b7d43fcde (127.0.192.195:43655)
I20260621 07:49:41.227962 15005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:41.224125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:32937 (local address 127.0.192.193:36707)
0621 07:49:41.224300 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:41.224305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:41.224386 (+ 81us) server_negotiation.cc:407] Connection header received
0621 07:49:41.224750 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:41.224752 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:41.224794 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:41.224891 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:49:41.225193 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:41.225950 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:41.226988 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:41.227383 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:41.227501 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:41.227549 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:41.227739 (+ 190us) server_negotiation.cc:299] Negotiation successful
0621 07:49:41.227800 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260621 07:49:41.227983 15019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:41.224047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:36707 (local address 127.0.192.194:32937)
0621 07:49:41.224300 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:41.224313 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:49:41.224436 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:49:41.224900 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:49:41.224903 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:49:41.224921 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:49:41.225047 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:41.225052 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:41.226054 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:41.226058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:41.226867 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:41.226875 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:41.227496 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:41.227500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:41.227615 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:41.227633 (+ 18us) client_negotiation.cc:769] Sending connection context
0621 07:49:41.227724 (+ 91us) client_negotiation.cc:240] Negotiation successful
0621 07:49:41.227790 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":28,"thread_start_us":114,"threads_started":1}
I20260621 07:49:41.228140 15009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:41.224628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:45153 (local address 127.0.192.195:43655)
0621 07:49:41.224787 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:41.224793 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:41.224808 (+ 15us) server_negotiation.cc:407] Connection header received
0621 07:49:41.224974 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:41.224977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:41.225022 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:41.225096 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:49:41.225395 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:41.226038 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:41.227105 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:41.227497 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:41.227646 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:41.227695 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:41.227924 (+ 229us) server_negotiation.cc:299] Negotiation successful
0621 07:49:41.228002 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 07:49:41.228323 15020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:41.224458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.195:43655 (local address 127.0.192.194:45153)
0621 07:49:41.224717 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:41.224731 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:49:41.224869 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:49:41.225126 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:49:41.225131 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:49:41.225152 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:49:41.225278 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:41.225284 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:41.226159 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:41.226163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:41.226967 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:41.226976 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:41.227638 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:41.227641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:41.227766 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:41.227784 (+ 18us) client_negotiation.cc:769] Sending connection context
0621 07:49:41.227909 (+ 125us) client_negotiation.cc:240] Negotiation successful
0621 07:49:41.227990 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":16,"thread_start_us":81,"threads_started":1}
I20260621 07:49:41.228632 14679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1fdcb1d8f474d9b9c911ad7015f9387" candidate_uuid: "a0f1558e9a154c5c9f2c171095b2d466" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74936c7336d24fd3b3070e3ff5b2105b" is_pre_election: true
I20260621 07:49:41.228948 14679 raft_consensus.cc:2454] T b1fdcb1d8f474d9b9c911ad7015f9387 P 74936c7336d24fd3b3070e3ff5b2105b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0f1558e9a154c5c9f2c171095b2d466 in term 0.
I20260621 07:49:41.228942 14947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1fdcb1d8f474d9b9c911ad7015f9387" candidate_uuid: "a0f1558e9a154c5c9f2c171095b2d466" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee318ad5e75847f49a88290b7d43fcde" is_pre_election: true
I20260621 07:49:41.229205 14947 raft_consensus.cc:2454] T b1fdcb1d8f474d9b9c911ad7015f9387 P ee318ad5e75847f49a88290b7d43fcde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0f1558e9a154c5c9f2c171095b2d466 in term 0.
I20260621 07:49:41.229403 14748 leader_election.cc:302] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74936c7336d24fd3b3070e3ff5b2105b, a0f1558e9a154c5c9f2c171095b2d466; no voters:
I20260621 07:49:41.229643 15015 raft_consensus.cc:2781] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:49:41.229723 15015 raft_consensus.cc:482] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:49:41.229772 15015 raft_consensus.cc:3037] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:49:41.230690 15015 raft_consensus.cc:504] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74936c7336d24fd3b3070e3ff5b2105b" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 36707 } } peers { permanent_uuid: "ee318ad5e75847f49a88290b7d43fcde" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43655 } } peers { permanent_uuid: "a0f1558e9a154c5c9f2c171095b2d466" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 36799 } }
I20260621 07:49:41.230973 15015 leader_election.cc:288] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466 [CANDIDATE]: Term 1 election: Requested vote from peers 74936c7336d24fd3b3070e3ff5b2105b (127.0.192.193:36707), ee318ad5e75847f49a88290b7d43fcde (127.0.192.195:43655)
I20260621 07:49:41.231297 14679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1fdcb1d8f474d9b9c911ad7015f9387" candidate_uuid: "a0f1558e9a154c5c9f2c171095b2d466" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74936c7336d24fd3b3070e3ff5b2105b"
I20260621 07:49:41.231333 14947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1fdcb1d8f474d9b9c911ad7015f9387" candidate_uuid: "a0f1558e9a154c5c9f2c171095b2d466" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee318ad5e75847f49a88290b7d43fcde"
I20260621 07:49:41.231400 14679 raft_consensus.cc:3037] T b1fdcb1d8f474d9b9c911ad7015f9387 P 74936c7336d24fd3b3070e3ff5b2105b [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:49:41.231495 14947 raft_consensus.cc:3037] T b1fdcb1d8f474d9b9c911ad7015f9387 P ee318ad5e75847f49a88290b7d43fcde [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:49:41.232383 14679 raft_consensus.cc:2454] T b1fdcb1d8f474d9b9c911ad7015f9387 P 74936c7336d24fd3b3070e3ff5b2105b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0f1558e9a154c5c9f2c171095b2d466 in term 1.
I20260621 07:49:41.232518 14947 raft_consensus.cc:2454] T b1fdcb1d8f474d9b9c911ad7015f9387 P ee318ad5e75847f49a88290b7d43fcde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0f1558e9a154c5c9f2c171095b2d466 in term 1.
I20260621 07:49:41.232687 14748 leader_election.cc:302] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74936c7336d24fd3b3070e3ff5b2105b, a0f1558e9a154c5c9f2c171095b2d466; no voters:
I20260621 07:49:41.232836 15015 raft_consensus.cc:2781] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:49:41.233026 15015 raft_consensus.cc:686] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466 [term 1 LEADER]: Becoming Leader. State: Replica: a0f1558e9a154c5c9f2c171095b2d466, State: Running, Role: LEADER
I20260621 07:49:41.233294 15015 consensus_queue.cc:237] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74936c7336d24fd3b3070e3ff5b2105b" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 36707 } } peers { permanent_uuid: "ee318ad5e75847f49a88290b7d43fcde" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43655 } } peers { permanent_uuid: "a0f1558e9a154c5c9f2c171095b2d466" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 36799 } }
I20260621 07:49:41.235916 14534 catalog_manager.cc:5495] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0f1558e9a154c5c9f2c171095b2d466 (127.0.192.194). New cstate: current_term: 1 leader_uuid: "a0f1558e9a154c5c9f2c171095b2d466" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74936c7336d24fd3b3070e3ff5b2105b" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 36707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee318ad5e75847f49a88290b7d43fcde" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0f1558e9a154c5c9f2c171095b2d466" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 36799 } health_report { overall_health: HEALTHY } } }
I20260621 07:49:41.321198 15008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:41.317699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45050 (local address 127.0.192.194:36799)
0621 07:49:41.317835 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:41.317839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:41.317853 (+ 14us) server_negotiation.cc:407] Connection header received
0621 07:49:41.318006 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:41.318009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:41.318062 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:41.318179 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:49:41.318419 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:41.318895 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:41.319736 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:41.319947 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:41.320062 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:41.320370 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:49:41.320798 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:49:41.320923 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:41.321003 (+ 80us) server_negotiation.cc:299] Negotiation successful
0621 07:49:41.321054 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 07:49:41.326531 14679 raft_consensus.cc:1261] T b1fdcb1d8f474d9b9c911ad7015f9387 P 74936c7336d24fd3b3070e3ff5b2105b [term 1 FOLLOWER]: Refusing update from remote peer a0f1558e9a154c5c9f2c171095b2d466: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:49:41.326531 14947 raft_consensus.cc:1261] T b1fdcb1d8f474d9b9c911ad7015f9387 P ee318ad5e75847f49a88290b7d43fcde [term 1 FOLLOWER]: Refusing update from remote peer a0f1558e9a154c5c9f2c171095b2d466: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:49:41.327247 15021 consensus_queue.cc:1035] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74936c7336d24fd3b3070e3ff5b2105b" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 36707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:49:41.327394 15015 consensus_queue.cc:1035] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee318ad5e75847f49a88290b7d43fcde" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:49:41.335625 15026 mvcc.cc:205] Tried to move back new op lower bound from 7299187430707179520 to 7299187430333935616. Current Snapshot: MvccSnapshot[applied={T|T < 7299187430707179520}]
I20260621 07:49:41.342985 14532 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:49:41.343106 14532 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:49:41.345211 14532 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a: Sending DeleteTablet for 3 replicas of tablet b1fdcb1d8f474d9b9c911ad7015f9387
I20260621 07:49:41.345927 14659 tablet_service.cc:1514] Processing DeleteTablet for tablet b1fdcb1d8f474d9b9c911ad7015f9387 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:49:41 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:51806
I20260621 07:49:41.345934 14793 tablet_service.cc:1514] Processing DeleteTablet for tablet b1fdcb1d8f474d9b9c911ad7015f9387 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:49:41 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:45044
I20260621 07:49:41.345984 14927 tablet_service.cc:1514] Processing DeleteTablet for tablet b1fdcb1d8f474d9b9c911ad7015f9387 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:49:41 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:58034
I20260621 07:49:41.346309 15034 tablet_replica.cc:330] T b1fdcb1d8f474d9b9c911ad7015f9387 P ee318ad5e75847f49a88290b7d43fcde: stopping tablet replica
I20260621 07:49:41.346365 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 14596
I20260621 07:49:41.346594 15034 raft_consensus.cc:2229] T b1fdcb1d8f474d9b9c911ad7015f9387 P ee318ad5e75847f49a88290b7d43fcde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:49:41.347245 15034 raft_consensus.cc:2258] T b1fdcb1d8f474d9b9c911ad7015f9387 P ee318ad5e75847f49a88290b7d43fcde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:49:41.348477 15034 ts_tablet_manager.cc:1905] T b1fdcb1d8f474d9b9c911ad7015f9387 P ee318ad5e75847f49a88290b7d43fcde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:49:41.351536 15034 ts_tablet_manager.cc:1918] T b1fdcb1d8f474d9b9c911ad7015f9387 P ee318ad5e75847f49a88290b7d43fcde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:49:41.351634 15034 log.cc:1196] T b1fdcb1d8f474d9b9c911ad7015f9387 P ee318ad5e75847f49a88290b7d43fcde: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/wal/wals/b1fdcb1d8f474d9b9c911ad7015f9387
I20260621 07:49:41.351934 15034 ts_tablet_manager.cc:1939] T b1fdcb1d8f474d9b9c911ad7015f9387 P ee318ad5e75847f49a88290b7d43fcde: Deleting consensus metadata
I20260621 07:49:41.352733 15035 tablet_replica.cc:330] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466: stopping tablet replica
I20260621 07:49:41.353037 15035 raft_consensus.cc:2229] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466 [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:49:41.353055 14520 catalog_manager.cc:4841] TS ee318ad5e75847f49a88290b7d43fcde (127.0.192.195:43655): tablet b1fdcb1d8f474d9b9c911ad7015f9387 (table test-table [id=9093818f183247cc8c6da0b1195984ac]) successfully deleted
I20260621 07:49:41.353456 15035 raft_consensus.cc:2258] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:49:41.354538 15035 ts_tablet_manager.cc:1905] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260621 07:49:41.354707 14520 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.192.193:36707 (error 108)
I20260621 07:49:41.354898 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 14730
I20260621 07:49:41.356043 15006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:41.355760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:36707 (local address 127.0.0.1:51818)
0621 07:49:41.355913 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:41.355969 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.193:36707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260621 07:49:41.356181 14520 catalog_manager.cc:4584] TS 74936c7336d24fd3b3070e3ff5b2105b (127.0.192.193:36707): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b1fdcb1d8f474d9b9c911ad7015f9387: Network error: Client connection negotiation failed: client connection to 127.0.192.193:36707: connect: Connection refused (error 111)
I20260621 07:49:41.361158 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 14864
I20260621 07:49:41.361295 15006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:41.361029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:36799 (local address 127.0.0.1:45052)
0621 07:49:41.361147 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:41.361200 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.194:36799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260621 07:49:41.366078 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 14502
I20260621 07:49:41.371826 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:42677
--webserver_interface=127.0.192.254
--webserver_port=41761
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:36019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:42677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:49:41.467633 15038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:49:41.467868 15038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:49:41.467944 15038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:49:41.471725 15038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:49:41.471917 15038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:49:41.472021 15038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:49:41.472062 15038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:49:41.472098 15038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:49:41.476862 15038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:36019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:42677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:42677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=41761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:49:41.477703 15038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:49:41.478533 15038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:49:41.489720 15047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:49:41.489727 15044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:49:41.489751 15045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:49:41.490020 15038 server_base.cc:1034] running on GCE node
I20260621 07:49:41.490562 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:49:41.491539 15038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:49:41.492722 15038 hybrid_clock.cc:648] HybridClock initialized: now 1782028181492690 us; error 46 us; skew 500 ppm
Jun 21 07:49:41 dist-test-slave-lzx3 krb5kdc[14489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:49:41.499858 15038 init.cc:377] Logged in from keytab as kudu/127.0.192.254@KRBTEST.COM (short username kudu)
I20260621 07:49:41.500634 15038 webserver.cc:460] Webserver started at http://127.0.192.254:41761/ using document root <none> and password file <none>
I20260621 07:49:41.501117 15038 fs_manager.cc:359] Metadata directory not provided
I20260621 07:49:41.501209 15038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:49:41.504495 15038 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260621 07:49:41.506529 15054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:41.507555 15038 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260621 07:49:41.507688 15038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "814aaaa5c60c4ca6a09cde0649decd1a"
format_stamp: "Formatted at 2026-06-21 07:49:40 on dist-test-slave-lzx3"
server_key: "514c1a28bb9e41ac7f516823c4ee3437"
server_key_iv: "9eddee813aeb52d38b7a1e12ca8afa73"
server_key_version: "encryptionkey@0"
I20260621 07:49:41.508198 15038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:49:41.542339 15038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:49:41.544716 15038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:49:41.544906 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:49:41.551208 15038 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:42677
I20260621 07:49:41.551491 15106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:42677 every 8 connection(s)
I20260621 07:49:41.552135 15038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:49:41.555801 15107 sys_catalog.cc:263] Verifying existing consensus state
I20260621 07:49:41.556931 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 15038
I20260621 07:49:41.557322 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.193:36707
--local_ip_for_outbound_sockets=127.0.192.193
--tserver_master_addrs=127.0.192.254:42677
--webserver_port=46859
--webserver_interface=127.0.192.193
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:36019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:49:41.557394 15107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a: Bootstrap starting.
I20260621 07:49:41.572389 15107 log.cc:824] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a: Log is configured to *not* fsync() on all Append() calls
I20260621 07:49:41.580152 15107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 07:49:41.580495 15107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a: Bootstrap complete.
I20260621 07:49:41.584255 15107 raft_consensus.cc:348] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814aaaa5c60c4ca6a09cde0649decd1a" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 42677 } }
I20260621 07:49:41.584661 15107 raft_consensus.cc:729] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 814aaaa5c60c4ca6a09cde0649decd1a, State: Initialized, Role: FOLLOWER
I20260621 07:49:41.585139 15107 consensus_queue.cc:260] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814aaaa5c60c4ca6a09cde0649decd1a" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 42677 } }
I20260621 07:49:41.585247 15107 raft_consensus.cc:388] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:49:41.585331 15107 raft_consensus.cc:482] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:49:41.585440 15107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [term 1 FOLLOWER]: Advancing to term 2
I20260621 07:49:41.587196 15107 raft_consensus.cc:504] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814aaaa5c60c4ca6a09cde0649decd1a" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 42677 } }
I20260621 07:49:41.587510 15107 leader_election.cc:302] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 814aaaa5c60c4ca6a09cde0649decd1a; no voters:
I20260621 07:49:41.587806 15107 leader_election.cc:288] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260621 07:49:41.587888 15111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [term 2 FOLLOWER]: Leader election won for term 2
I20260621 07:49:41.588129 15111 raft_consensus.cc:686] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [term 2 LEADER]: Becoming Leader. State: Replica: 814aaaa5c60c4ca6a09cde0649decd1a, State: Running, Role: LEADER
I20260621 07:49:41.588404 15111 consensus_queue.cc:237] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814aaaa5c60c4ca6a09cde0649decd1a" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 42677 } }
I20260621 07:49:41.588994 15107 sys_catalog.cc:564] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:49:41.589238 15111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 814aaaa5c60c4ca6a09cde0649decd1a. Latest consensus state: current_term: 2 leader_uuid: "814aaaa5c60c4ca6a09cde0649decd1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814aaaa5c60c4ca6a09cde0649decd1a" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 42677 } } }
I20260621 07:49:41.589298 15112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "814aaaa5c60c4ca6a09cde0649decd1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814aaaa5c60c4ca6a09cde0649decd1a" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 42677 } } }
I20260621 07:49:41.589525 15112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [sys.catalog]: This master's current role is: LEADER
I20260621 07:49:41.589345 15111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [sys.catalog]: This master's current role is: LEADER
I20260621 07:49:41.589937 15115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:49:41.594633 15115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:49:41.595310 15115 catalog_manager.cc:1259] Loaded cluster ID: 138f75245e30495293a83cb39ad5ff64
I20260621 07:49:41.595384 15115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:49:41.596428 15115 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260621 07:49:41.599115 15115 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260621 07:49:41.603179 15115 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:49:41.603847 15115 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a: Loaded TSK: 0
I20260621 07:49:41.604799 15115 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 07:49:41.670794 15109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:49:41.671032 15109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:49:41.671111 15109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:49:41.677353 15109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:49:41.677417 15109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:49:41.677520 15109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.193
I20260621 07:49:41.682749 15109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:36019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.193:36707
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.192.193
--webserver_port=46859
--tserver_master_addrs=127.0.192.254:42677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.193
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:49:41.683632 15109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:49:41.684458 15109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:49:41.693084 15134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:49:41.693090 15135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:49:41.693094 15137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:49:41.693758 15109 server_base.cc:1034] running on GCE node
I20260621 07:49:41.694149 15109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:49:41.694766 15109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:49:41.695960 15109 hybrid_clock.cc:648] HybridClock initialized: now 1782028181695932 us; error 48 us; skew 500 ppm
Jun 21 07:49:41 dist-test-slave-lzx3 krb5kdc[14489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:49:41.698845 15109 init.cc:377] Logged in from keytab as kudu/127.0.192.193@KRBTEST.COM (short username kudu)
I20260621 07:49:41.699608 15109 webserver.cc:460] Webserver started at http://127.0.192.193:46859/ using document root <none> and password file <none>
I20260621 07:49:41.700105 15109 fs_manager.cc:359] Metadata directory not provided
I20260621 07:49:41.700172 15109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:49:41.703917 15109 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:49:41.705979 15144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:41.707126 15109 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 07:49:41.707252 15109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/wal
uuid: "74936c7336d24fd3b3070e3ff5b2105b"
format_stamp: "Formatted at 2026-06-21 07:49:40 on dist-test-slave-lzx3"
server_key: "619b4bc1ac893ba2ce926b0f86126a56"
server_key_iv: "b462ad764d7f3735f8c06fda5a8ac4e0"
server_key_version: "encryptionkey@0"
I20260621 07:49:41.707695 15109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:49:41.733150 15109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:49:41.735994 15109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:49:41.736184 15109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:49:41.736732 15109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:49:41.737656 15151 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260621 07:49:41.741044 15109 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260621 07:49:41.741106 15109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260621 07:49:41.741190 15109 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260621 07:49:41.742677 15109 ts_tablet_manager.cc:610] Registered 1 tablets
I20260621 07:49:41.742749 15109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260621 07:49:41.743032 15151 tablet_bootstrap.cc:492] T b1fdcb1d8f474d9b9c911ad7015f9387 P 74936c7336d24fd3b3070e3ff5b2105b: Bootstrap starting.
I20260621 07:49:41.753675 15109 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.193:36707
I20260621 07:49:41.753749 15258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.193:36707 every 8 connection(s)
I20260621 07:49:41.754570 15109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
I20260621 07:49:41.761488 15151 log.cc:824] T b1fdcb1d8f474d9b9c911ad7015f9387 P 74936c7336d24fd3b3070e3ff5b2105b: Log is configured to *not* fsync() on all Append() calls
I20260621 07:49:41.763247 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 15109
I20260621 07:49:41.763793 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.194:36799
--local_ip_for_outbound_sockets=127.0.192.194
--tserver_master_addrs=127.0.192.254:42677
--webserver_port=33447
--webserver_interface=127.0.192.194
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:36019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 07:49:41 dist-test-slave-lzx3 krb5kdc[14489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:49:41.768726 15151 tablet_bootstrap.cc:492] T b1fdcb1d8f474d9b9c911ad7015f9387 P 74936c7336d24fd3b3070e3ff5b2105b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 07:49:41.769084 15151 tablet_bootstrap.cc:492] T b1fdcb1d8f474d9b9c911ad7015f9387 P 74936c7336d24fd3b3070e3ff5b2105b: Bootstrap complete.
I20260621 07:49:41.770320 15151 ts_tablet_manager.cc:1397] T b1fdcb1d8f474d9b9c911ad7015f9387 P 74936c7336d24fd3b3070e3ff5b2105b: Time spent bootstrapping tablet: real 0.028s user 0.018s sys 0.008s
I20260621 07:49:41.771987 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:41.756725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:50481 (local address 127.0.192.254:42677)
0621 07:49:41.757141 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:41.757159 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:41.757486 (+ 327us) server_negotiation.cc:407] Connection header received
0621 07:49:41.758592 (+ 1106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:41.758606 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:41.758981 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:41.759360 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:49:41.760308 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:41.761650 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:41.762774 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:41.763436 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:41.763593 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:41.766422 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:49:41.766428 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:49:41.766442 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:49:41.766473 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:49:41.768495 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:41.769163 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:41.769168 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:41.769173 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:41.769303 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:41.769736 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:41.769742 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:41.769746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:41.770177 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:49:41.770300 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:41.770978 (+ 678us) server_negotiation.cc:299] Negotiation successful
0621 07:49:41.771231 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":146,"threads_started":1}
I20260621 07:49:41.772112 15262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:41.756627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.193:50481)
0621 07:49:41.757264 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:41.757312 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0621 07:49:41.758064 (+ 752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:49:41.759392 (+ 1328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:49:41.759401 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:49:41.759729 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:49:41.760138 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:41.760151 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:41.761788 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:41.761790 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:41.762649 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:41.762659 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:41.763606 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:41.763612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:41.763797 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:41.764429 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:49:41.764455 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:49:41.766227 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:49:41.768631 (+ 2404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:41.768639 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:41.768655 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:41.769016 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:41.769421 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:41.769426 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:41.769430 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:41.769626 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:41.770303 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:49:41.770311 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:49:41.770611 (+ 300us) client_negotiation.cc:769] Sending connection context
0621 07:49:41.770849 (+ 238us) client_negotiation.cc:240] Negotiation successful
0621 07:49:41.771184 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":138,"threads_started":1}
I20260621 07:49:41.772814 15151 raft_consensus.cc:348] T b1fdcb1d8f474d9b9c911ad7015f9387 P 74936c7336d24fd3b3070e3ff5b2105b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74936c7336d24fd3b3070e3ff5b2105b" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 36707 } } peers { permanent_uuid: "ee318ad5e75847f49a88290b7d43fcde" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43655 } } peers { permanent_uuid: "a0f1558e9a154c5c9f2c171095b2d466" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 36799 } }
I20260621 07:49:41.773239 15151 raft_consensus.cc:729] T b1fdcb1d8f474d9b9c911ad7015f9387 P 74936c7336d24fd3b3070e3ff5b2105b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74936c7336d24fd3b3070e3ff5b2105b, State: Initialized, Role: FOLLOWER
I20260621 07:49:41.773655 15151 consensus_queue.cc:260] T b1fdcb1d8f474d9b9c911ad7015f9387 P 74936c7336d24fd3b3070e3ff5b2105b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74936c7336d24fd3b3070e3ff5b2105b" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 36707 } } peers { permanent_uuid: "ee318ad5e75847f49a88290b7d43fcde" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43655 } } peers { permanent_uuid: "a0f1558e9a154c5c9f2c171095b2d466" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 36799 } }
I20260621 07:49:41.774432 15151 ts_tablet_manager.cc:1428] T b1fdcb1d8f474d9b9c911ad7015f9387 P 74936c7336d24fd3b3070e3ff5b2105b: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260621 07:49:41.774569 15259 heartbeater.cc:347] Connected to a master server at 127.0.192.254:42677
I20260621 07:49:41.774777 15259 heartbeater.cc:464] Registering TS with master...
I20260621 07:49:41.775331 15259 heartbeater.cc:511] Master 127.0.192.254:42677 requested a full tablet report, sending...
I20260621 07:49:41.777740 15071 ts_manager.cc:194] Registered new tserver with Master: 74936c7336d24fd3b3070e3ff5b2105b (127.0.192.193:36707)
I20260621 07:49:41.780738 15071 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.193@KRBTEST.COM'} at 127.0.192.193:50481
I20260621 07:49:41.782953 15259 heartbeater.cc:503] Master 127.0.192.254:42677 was elected leader, sending a full tablet report...
Jun 21 07:49:41 dist-test-slave-lzx3 krb5kdc[14489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for kudu/127.0.192.193@KRBTEST.COM
I20260621 07:49:41.790148 15266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:41.780098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:36707 (local address 127.0.0.1:51832)
0621 07:49:41.780468 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:41.780510 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 07:49:41.780679 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:49:41.781177 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:49:41.781185 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:49:41.781619 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:49:41.781741 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:41.781749 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:41.782964 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:41.782968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:41.783900 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:41.783911 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:41.784493 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:41.784497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:41.784638 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:41.785051 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:49:41.785067 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:49:41.786607 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:49:41.788364 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:41.788382 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:41.788387 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:41.788750 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:41.789020 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:41.789023 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:41.789024 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:41.789119 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:41.789519 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:49:41.789525 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:49:41.789648 (+ 123us) client_negotiation.cc:769] Sending connection context
0621 07:49:41.789816 (+ 168us) client_negotiation.cc:240] Negotiation successful
0621 07:49:41.789984 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":87,"threads_started":1}
I20260621 07:49:41.790192 15267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:41.780175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51832 (local address 127.0.192.193:36707)
0621 07:49:41.780729 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:41.780738 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:41.780776 (+ 38us) server_negotiation.cc:407] Connection header received
0621 07:49:41.780831 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:41.780835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:41.781018 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:41.781137 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:49:41.781861 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:41.782837 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:41.784024 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:41.784387 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:41.784462 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:41.786741 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:49:41.786749 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:49:41.786757 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:49:41.786777 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:49:41.788252 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:41.788849 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:41.788853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:41.788854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:41.788930 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:41.789209 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:41.789211 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:41.789212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:41.789391 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:49:41.789536 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:41.789913 (+ 377us) server_negotiation.cc:299] Negotiation successful
0621 07:49:41.790048 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":468,"thread_start_us":121,"threads_started":1}
I20260621 07:49:41.791096 15193 tablet_service.cc:1514] Processing DeleteTablet for tablet b1fdcb1d8f474d9b9c911ad7015f9387 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:49:41 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:51832
I20260621 07:49:41.791936 15268 tablet_replica.cc:330] stopping tablet replica
I20260621 07:49:41.792176 15268 raft_consensus.cc:2229] T b1fdcb1d8f474d9b9c911ad7015f9387 P 74936c7336d24fd3b3070e3ff5b2105b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:49:41.792390 15268 raft_consensus.cc:2258] T b1fdcb1d8f474d9b9c911ad7015f9387 P 74936c7336d24fd3b3070e3ff5b2105b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:49:41.793165 15268 ts_tablet_manager.cc:1905] T b1fdcb1d8f474d9b9c911ad7015f9387 P 74936c7336d24fd3b3070e3ff5b2105b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:49:41.796222 15268 ts_tablet_manager.cc:1918] T b1fdcb1d8f474d9b9c911ad7015f9387 P 74936c7336d24fd3b3070e3ff5b2105b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:49:41.796317 15268 log.cc:1196] T b1fdcb1d8f474d9b9c911ad7015f9387 P 74936c7336d24fd3b3070e3ff5b2105b: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/wal/wals/b1fdcb1d8f474d9b9c911ad7015f9387
I20260621 07:49:41.796608 15268 ts_tablet_manager.cc:1939] T b1fdcb1d8f474d9b9c911ad7015f9387 P 74936c7336d24fd3b3070e3ff5b2105b: Deleting consensus metadata
I20260621 07:49:41.797998 15057 catalog_manager.cc:4841] TS 74936c7336d24fd3b3070e3ff5b2105b (127.0.192.193:36707): tablet b1fdcb1d8f474d9b9c911ad7015f9387 (table test-table [id=9093818f183247cc8c6da0b1195984ac]) successfully deleted
W20260621 07:49:41.873013 15264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:49:41.873260 15264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:49:41.873339 15264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:49:41.877290 15264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:49:41.877368 15264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:49:41.877455 15264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.194
I20260621 07:49:41.882170 15264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:36019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.194:36799
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.192.194
--webserver_port=33447
--tserver_master_addrs=127.0.192.254:42677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.194
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:49:41.883045 15264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:49:41.883898 15264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:49:41.892606 15273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:49:41.892727 15264 server_base.cc:1034] running on GCE node
W20260621 07:49:41.892582 15274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:49:41.892597 15276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:49:41.893203 15264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:49:41.893731 15264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:49:41.894909 15264 hybrid_clock.cc:648] HybridClock initialized: now 1782028181894897 us; error 31 us; skew 500 ppm
Jun 21 07:49:41 dist-test-slave-lzx3 krb5kdc[14489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:49:41.897428 15264 init.cc:377] Logged in from keytab as kudu/127.0.192.194@KRBTEST.COM (short username kudu)
I20260621 07:49:41.898205 15264 webserver.cc:460] Webserver started at http://127.0.192.194:33447/ using document root <none> and password file <none>
I20260621 07:49:41.898667 15264 fs_manager.cc:359] Metadata directory not provided
I20260621 07:49:41.898761 15264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:49:41.901963 15264 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260621 07:49:41.903935 15283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:41.905090 15264 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260621 07:49:41.905229 15264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/wal
uuid: "a0f1558e9a154c5c9f2c171095b2d466"
format_stamp: "Formatted at 2026-06-21 07:49:40 on dist-test-slave-lzx3"
server_key: "d9b79482b0666d353e0c86f31c4c0f49"
server_key_iv: "d4705c943e09bad2d1a0d6837cb9abd2"
server_key_version: "encryptionkey@0"
I20260621 07:49:41.905709 15264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:49:41.913872 15264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:49:41.916232 15264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:49:41.916435 15264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:49:41.917029 15264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:49:41.917968 15290 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260621 07:49:41.921299 15264 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260621 07:49:41.921376 15264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260621 07:49:41.921450 15264 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260621 07:49:41.922927 15264 ts_tablet_manager.cc:610] Registered 1 tablets
I20260621 07:49:41.923004 15264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260621 07:49:41.923254 15290 tablet_bootstrap.cc:492] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466: Bootstrap starting.
I20260621 07:49:41.928218 15290 log.cc:824] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466: Log is configured to *not* fsync() on all Append() calls
I20260621 07:49:41.933760 15264 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.194:36799
I20260621 07:49:41.933832 15398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.194:36799 every 8 connection(s)
I20260621 07:49:41.934523 15264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
I20260621 07:49:41.934954 15290 tablet_bootstrap.cc:492] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 07:49:41.935654 15290 tablet_bootstrap.cc:492] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466: Bootstrap complete.
I20260621 07:49:41.935948 15290 ts_tablet_manager.cc:1397] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20260621 07:49:41.938468 15290 raft_consensus.cc:348] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74936c7336d24fd3b3070e3ff5b2105b" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 36707 } } peers { permanent_uuid: "ee318ad5e75847f49a88290b7d43fcde" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43655 } } peers { permanent_uuid: "a0f1558e9a154c5c9f2c171095b2d466" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 36799 } }
I20260621 07:49:41.938807 15290 raft_consensus.cc:729] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0f1558e9a154c5c9f2c171095b2d466, State: Initialized, Role: FOLLOWER
I20260621 07:49:41.939199 15290 consensus_queue.cc:260] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74936c7336d24fd3b3070e3ff5b2105b" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 36707 } } peers { permanent_uuid: "ee318ad5e75847f49a88290b7d43fcde" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43655 } } peers { permanent_uuid: "a0f1558e9a154c5c9f2c171095b2d466" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 36799 } }
I20260621 07:49:41.939925 15290 ts_tablet_manager.cc:1428] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260621 07:49:41.940598 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 15264
I20260621 07:49:41.940924 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.195:43655
--local_ip_for_outbound_sockets=127.0.192.195
--tserver_master_addrs=127.0.192.254:42677
--webserver_port=38937
--webserver_interface=127.0.192.195
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:36019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 07:49:41 dist-test-slave-lzx3 krb5kdc[14489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:49:41.947978 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:41.936194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:42217 (local address 127.0.192.254:42677)
0621 07:49:41.936339 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:41.936344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:41.936947 (+ 603us) server_negotiation.cc:407] Connection header received
0621 07:49:41.938034 (+ 1087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:41.938038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:41.938102 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:41.938182 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:49:41.939287 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:41.939812 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:41.940974 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:41.941261 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:41.941375 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:41.944406 (+ 3031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:49:41.944410 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:49:41.944413 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:49:41.944445 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:49:41.945920 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:41.946457 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:41.946460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:41.946463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:41.946554 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:41.946921 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:41.946924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:41.946926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:41.947100 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:49:41.947204 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:41.947664 (+ 460us) server_negotiation.cc:299] Negotiation successful
0621 07:49:41.947812 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 07:49:41.948760 15401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:41.936309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.194:42217)
0621 07:49:41.936792 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:41.936850 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0621 07:49:41.937844 (+ 994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:49:41.938336 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:49:41.938342 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:49:41.938624 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:49:41.939144 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:41.939158 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:41.939951 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:41.939955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:41.940858 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:41.940867 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:41.941371 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:41.941375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:41.941544 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:41.942397 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:49:41.942424 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:49:41.944221 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:49:41.946048 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:41.946054 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:41.946066 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:41.946334 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:41.946658 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:41.946661 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:41.946664 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:41.946818 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:41.947203 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:49:41.947209 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:49:41.947423 (+ 214us) client_negotiation.cc:769] Sending connection context
0621 07:49:41.947594 (+ 171us) client_negotiation.cc:240] Negotiation successful
0621 07:49:41.947888 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":143,"threads_started":1}
I20260621 07:49:41.949975 15399 heartbeater.cc:347] Connected to a master server at 127.0.192.254:42677
I20260621 07:49:41.950186 15399 heartbeater.cc:464] Registering TS with master...
I20260621 07:49:41.950614 15399 heartbeater.cc:511] Master 127.0.192.254:42677 requested a full tablet report, sending...
I20260621 07:49:41.952172 15071 ts_manager.cc:194] Registered new tserver with Master: a0f1558e9a154c5c9f2c171095b2d466 (127.0.192.194:36799)
I20260621 07:49:41.953313 15071 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.194@KRBTEST.COM'} at 127.0.192.194:42217
I20260621 07:49:41.955186 15399 heartbeater.cc:503] Master 127.0.192.254:42677 was elected leader, sending a full tablet report...
Jun 21 07:49:41 dist-test-slave-lzx3 krb5kdc[14489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for kudu/127.0.192.194@KRBTEST.COM
I20260621 07:49:41.961939 15404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:41.953152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45054 (local address 127.0.192.194:36799)
0621 07:49:41.953519 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:41.953546 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:41.953567 (+ 21us) server_negotiation.cc:407] Connection header received
0621 07:49:41.953628 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:41.953632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:41.953781 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:41.953927 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:49:41.954294 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:41.955260 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:41.956406 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:41.957018 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:41.957116 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:41.959103 (+ 1987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:49:41.959110 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:49:41.959118 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:49:41.959136 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:49:41.960441 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:41.960892 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:41.960896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:41.960897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:41.960963 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:41.961207 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:41.961209 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:41.961210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:41.961398 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:49:41.961488 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:41.961730 (+ 242us) server_negotiation.cc:299] Negotiation successful
0621 07:49:41.961831 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":84,"threads_started":1}
I20260621 07:49:41.961939 15266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:41.953064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:36799 (local address 127.0.0.1:45054)
0621 07:49:41.953238 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:41.953253 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:49:41.953320 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:49:41.953931 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:49:41.953934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:49:41.954092 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:49:41.954176 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:41.954181 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:41.955370 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:41.955373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:41.956237 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:41.956250 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:41.957141 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:41.957144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:41.957208 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:41.957620 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:49:41.957686 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:49:41.958986 (+ 1300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:49:41.960541 (+ 1555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:41.960545 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:41.960548 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:41.960792 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:41.961046 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:41.961049 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:41.961051 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:41.961131 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:41.961494 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:49:41.961498 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:49:41.961584 (+ 86us) client_negotiation.cc:769] Sending connection context
0621 07:49:41.961686 (+ 102us) client_negotiation.cc:240] Negotiation successful
0621 07:49:41.961802 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260621 07:49:41.963021 15332 tablet_service.cc:1514] Processing DeleteTablet for tablet b1fdcb1d8f474d9b9c911ad7015f9387 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:49:41 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:45054
I20260621 07:49:41.963825 15405 tablet_replica.cc:330] stopping tablet replica
I20260621 07:49:41.964079 15405 raft_consensus.cc:2229] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:49:41.964285 15405 raft_consensus.cc:2258] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:49:41.965214 15405 ts_tablet_manager.cc:1905] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:49:41.969127 15405 ts_tablet_manager.cc:1918] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:49:41.969208 15405 log.cc:1196] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/wal/wals/b1fdcb1d8f474d9b9c911ad7015f9387
I20260621 07:49:41.969483 15405 ts_tablet_manager.cc:1939] T b1fdcb1d8f474d9b9c911ad7015f9387 P a0f1558e9a154c5c9f2c171095b2d466: Deleting consensus metadata
I20260621 07:49:41.970870 15056 catalog_manager.cc:4841] TS a0f1558e9a154c5c9f2c171095b2d466 (127.0.192.194:36799): tablet b1fdcb1d8f474d9b9c911ad7015f9387 (table test-table [id=9093818f183247cc8c6da0b1195984ac]) successfully deleted
W20260621 07:49:42.044164 15403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:49:42.044346 15403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:49:42.044379 15403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:49:42.048154 15403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:49:42.048200 15403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:49:42.048249 15403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.195
I20260621 07:49:42.052783 15403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:36019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.195:43655
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.192.195
--webserver_port=38937
--tserver_master_addrs=127.0.192.254:42677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.195
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:49:42.053593 15403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:49:42.054396 15403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:49:42.062682 15410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:49:42.062757 15403 server_base.cc:1034] running on GCE node
W20260621 07:49:42.062660 15411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:49:42.062680 15413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:49:42.063242 15403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:49:42.063741 15403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:49:42.064914 15403 hybrid_clock.cc:648] HybridClock initialized: now 1782028182064889 us; error 37 us; skew 500 ppm
Jun 21 07:49:42 dist-test-slave-lzx3 krb5kdc[14489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:49:42.067407 15403 init.cc:377] Logged in from keytab as kudu/127.0.192.195@KRBTEST.COM (short username kudu)
I20260621 07:49:42.068107 15403 webserver.cc:460] Webserver started at http://127.0.192.195:38937/ using document root <none> and password file <none>
I20260621 07:49:42.068521 15403 fs_manager.cc:359] Metadata directory not provided
I20260621 07:49:42.068629 15403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:49:42.071779 15403 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260621 07:49:42.073529 15420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:42.074532 15403 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 07:49:42.074612 15403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/wal
uuid: "ee318ad5e75847f49a88290b7d43fcde"
format_stamp: "Formatted at 2026-06-21 07:49:40 on dist-test-slave-lzx3"
server_key: "7f7ea3f47e3b338096e6055e96534c32"
server_key_iv: "e68ad8c496595a39d94c6e5694352ec9"
server_key_version: "encryptionkey@0"
I20260621 07:49:42.075016 15403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:49:42.105165 15403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:49:42.108111 15403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:49:42.108242 15403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:49:42.108853 15403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:49:42.109592 15403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:49:42.109638 15403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:42.109675 15403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:49:42.109733 15403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:42.118528 15403 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.195:43655
I20260621 07:49:42.118571 15533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.195:43655 every 8 connection(s)
I20260621 07:49:42.119446 15403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
I20260621 07:49:42.126993 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 15403
Jun 21 07:49:42 dist-test-slave-lzx3 krb5kdc[14489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:49:42.132091 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:42.121047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:50563 (local address 127.0.192.254:42677)
0621 07:49:42.121160 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:42.121163 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:42.121845 (+ 682us) server_negotiation.cc:407] Connection header received
0621 07:49:42.123061 (+ 1216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:42.123065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:42.123133 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:42.123246 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:49:42.124236 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:42.124819 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:42.125892 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:42.126110 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:42.126219 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:42.128690 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:49:42.128694 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:49:42.128697 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:49:42.128729 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:49:42.130221 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:42.130700 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:42.130703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:42.130705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:42.130788 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:42.131104 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:42.131108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:42.131110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:42.131276 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:49:42.131399 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:42.131827 (+ 428us) server_negotiation.cc:299] Negotiation successful
0621 07:49:42.131931 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260621 07:49:42.132897 15536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:42.121334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.195:50563)
0621 07:49:42.121706 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:42.121742 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 07:49:42.122868 (+ 1126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:49:42.123393 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:49:42.123398 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:49:42.123683 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:49:42.124099 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:42.124111 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:42.124934 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:42.124943 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:42.125758 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:42.125776 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:42.126250 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:42.126254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:42.126412 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:42.126906 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:49:42.126926 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:49:42.128505 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:49:42.130337 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:42.130341 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:42.130350 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:42.130595 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:42.130875 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:42.130879 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:42.130881 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:42.131006 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:42.131397 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:49:42.131403 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:49:42.131610 (+ 207us) client_negotiation.cc:769] Sending connection context
0621 07:49:42.131783 (+ 173us) client_negotiation.cc:240] Negotiation successful
0621 07:49:42.132038 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":93,"threads_started":1}
I20260621 07:49:42.133948 15534 heartbeater.cc:347] Connected to a master server at 127.0.192.254:42677
I20260621 07:49:42.134161 15534 heartbeater.cc:464] Registering TS with master...
I20260621 07:49:42.134547 15534 heartbeater.cc:511] Master 127.0.192.254:42677 requested a full tablet report, sending...
I20260621 07:49:42.135397 15538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:42.127584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41572 (local address 127.0.192.254:42677)
0621 07:49:42.127790 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:42.127794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:42.127926 (+ 132us) server_negotiation.cc:407] Connection header received
0621 07:49:42.128003 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:42.128006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:42.128045 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:42.128127 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:49:42.128909 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:42.129342 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:42.130550 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:42.130769 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:42.130864 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:42.132109 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:49:42.132112 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:49:42.132115 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:49:42.132138 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:49:42.133928 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:42.134322 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:42.134326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:42.134328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:42.134410 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:42.134670 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:42.134674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:42.134676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:42.134833 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:49:42.135020 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:42.135149 (+ 129us) server_negotiation.cc:299] Negotiation successful
0621 07:49:42.135247 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":63,"threads_started":1}
I20260621 07:49:42.135479 15071 ts_manager.cc:194] Registered new tserver with Master: ee318ad5e75847f49a88290b7d43fcde (127.0.192.195:43655)
I20260621 07:49:42.136166 15071 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.195@KRBTEST.COM'} at 127.0.192.195:50563
I20260621 07:49:42.137017 771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:49:42.140512 15538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:42.137522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41588 (local address 127.0.192.254:42677)
0621 07:49:42.137592 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:42.137597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:42.138123 (+ 526us) server_negotiation.cc:407] Connection header received
0621 07:49:42.138235 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:42.138237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:42.138298 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:42.138363 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:49:42.138474 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:42.138870 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:42.139564 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:42.139741 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:42.139804 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:42.140016 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:49:42.140283 (+ 267us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:49:42.140369 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:42.140415 (+ 46us) server_negotiation.cc:299] Negotiation successful
0621 07:49:42.140448 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260621 07:49:42.141024 15070 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:41588:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:49:42.143857 15070 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:49:42.150257 15193 tablet_service.cc:1467] Processing CreateTablet for tablet cdca2e1b1ce14d44a1868ecc7c4c2e37 (DEFAULT_TABLE table=test-table [id=1bd8fa3267654ed2995636a593379dac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:49:42.150398 15332 tablet_service.cc:1467] Processing CreateTablet for tablet cdca2e1b1ce14d44a1868ecc7c4c2e37 (DEFAULT_TABLE table=test-table [id=1bd8fa3267654ed2995636a593379dac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:49:42.150781 15193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdca2e1b1ce14d44a1868ecc7c4c2e37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:49:42.150779 15332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdca2e1b1ce14d44a1868ecc7c4c2e37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:49:42.153235 15151 tablet_bootstrap.cc:492] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b: Bootstrap starting.
I20260621 07:49:42.153235 15290 tablet_bootstrap.cc:492] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P a0f1558e9a154c5c9f2c171095b2d466: Bootstrap starting.
I20260621 07:49:42.153527 15266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:42.149475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.195:43655 (local address 127.0.0.1:58038)
0621 07:49:42.149664 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:42.149676 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:49:42.149748 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:49:42.150311 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:49:42.150313 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:49:42.150328 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:49:42.150398 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:42.150406 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:42.151542 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:42.151547 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:42.152233 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:42.152242 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:42.153197 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:42.153200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:42.153272 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:42.153304 (+ 32us) client_negotiation.cc:769] Sending connection context
0621 07:49:42.153378 (+ 74us) client_negotiation.cc:240] Negotiation successful
0621 07:49:42.153433 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134}
I20260621 07:49:42.153574 15543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:42.149619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58038 (local address 127.0.192.195:43655)
0621 07:49:42.149892 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:42.149905 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:42.149955 (+ 50us) server_negotiation.cc:407] Connection header received
0621 07:49:42.150010 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:42.150015 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:42.150169 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:42.150321 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:49:42.150478 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:42.151294 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:42.152343 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:42.153004 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:42.153192 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:42.153310 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:42.153400 (+ 90us) server_negotiation.cc:299] Negotiation successful
0621 07:49:42.153467 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":108,"threads_started":1}
I20260621 07:49:42.153995 15151 tablet_bootstrap.cc:654] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b: Neither blocks nor log segments found. Creating new log.
I20260621 07:49:42.153995 15290 tablet_bootstrap.cc:654] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P a0f1558e9a154c5c9f2c171095b2d466: Neither blocks nor log segments found. Creating new log.
I20260621 07:49:42.155423 15290 tablet_bootstrap.cc:492] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P a0f1558e9a154c5c9f2c171095b2d466: No bootstrap required, opened a new log
I20260621 07:49:42.155529 15290 ts_tablet_manager.cc:1397] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P a0f1558e9a154c5c9f2c171095b2d466: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260621 07:49:42.155737 15151 tablet_bootstrap.cc:492] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b: No bootstrap required, opened a new log
I20260621 07:49:42.155807 15151 ts_tablet_manager.cc:1397] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260621 07:49:42.155963 15290 raft_consensus.cc:348] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P a0f1558e9a154c5c9f2c171095b2d466 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee318ad5e75847f49a88290b7d43fcde" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43655 } } peers { permanent_uuid: "a0f1558e9a154c5c9f2c171095b2d466" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 36799 } } peers { permanent_uuid: "74936c7336d24fd3b3070e3ff5b2105b" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 36707 } }
I20260621 07:49:42.156081 15290 raft_consensus.cc:374] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P a0f1558e9a154c5c9f2c171095b2d466 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:49:42.156126 15290 raft_consensus.cc:729] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P a0f1558e9a154c5c9f2c171095b2d466 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0f1558e9a154c5c9f2c171095b2d466, State: Initialized, Role: FOLLOWER
I20260621 07:49:42.156186 15151 raft_consensus.cc:348] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee318ad5e75847f49a88290b7d43fcde" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43655 } } peers { permanent_uuid: "a0f1558e9a154c5c9f2c171095b2d466" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 36799 } } peers { permanent_uuid: "74936c7336d24fd3b3070e3ff5b2105b" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 36707 } }
I20260621 07:49:42.156311 15151 raft_consensus.cc:374] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:49:42.156239 15290 consensus_queue.cc:260] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P a0f1558e9a154c5c9f2c171095b2d466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee318ad5e75847f49a88290b7d43fcde" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43655 } } peers { permanent_uuid: "a0f1558e9a154c5c9f2c171095b2d466" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 36799 } } peers { permanent_uuid: "74936c7336d24fd3b3070e3ff5b2105b" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 36707 } }
I20260621 07:49:42.156361 15151 raft_consensus.cc:729] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74936c7336d24fd3b3070e3ff5b2105b, State: Initialized, Role: FOLLOWER
I20260621 07:49:42.156476 15151 consensus_queue.cc:260] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee318ad5e75847f49a88290b7d43fcde" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43655 } } peers { permanent_uuid: "a0f1558e9a154c5c9f2c171095b2d466" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 36799 } } peers { permanent_uuid: "74936c7336d24fd3b3070e3ff5b2105b" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 36707 } }
I20260621 07:49:42.156541 15290 ts_tablet_manager.cc:1428] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P a0f1558e9a154c5c9f2c171095b2d466: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260621 07:49:42.156787 15151 ts_tablet_manager.cc:1428] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260621 07:49:42.156347 15468 tablet_service.cc:1467] Processing CreateTablet for tablet cdca2e1b1ce14d44a1868ecc7c4c2e37 (DEFAULT_TABLE table=test-table [id=1bd8fa3267654ed2995636a593379dac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:49:42.157335 15468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdca2e1b1ce14d44a1868ecc7c4c2e37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:49:42.163481 15546 tablet_bootstrap.cc:492] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P ee318ad5e75847f49a88290b7d43fcde: Bootstrap starting.
I20260621 07:49:42.165438 15546 tablet_bootstrap.cc:654] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P ee318ad5e75847f49a88290b7d43fcde: Neither blocks nor log segments found. Creating new log.
I20260621 07:49:42.166123 15546 log.cc:824] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P ee318ad5e75847f49a88290b7d43fcde: Log is configured to *not* fsync() on all Append() calls
I20260621 07:49:42.167814 15546 tablet_bootstrap.cc:492] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P ee318ad5e75847f49a88290b7d43fcde: No bootstrap required, opened a new log
I20260621 07:49:42.167984 15546 ts_tablet_manager.cc:1397] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P ee318ad5e75847f49a88290b7d43fcde: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260621 07:49:42.170503 15546 raft_consensus.cc:348] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P ee318ad5e75847f49a88290b7d43fcde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee318ad5e75847f49a88290b7d43fcde" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43655 } } peers { permanent_uuid: "a0f1558e9a154c5c9f2c171095b2d466" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 36799 } } peers { permanent_uuid: "74936c7336d24fd3b3070e3ff5b2105b" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 36707 } }
I20260621 07:49:42.170686 15546 raft_consensus.cc:374] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P ee318ad5e75847f49a88290b7d43fcde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:49:42.170770 15546 raft_consensus.cc:729] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P ee318ad5e75847f49a88290b7d43fcde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee318ad5e75847f49a88290b7d43fcde, State: Initialized, Role: FOLLOWER
I20260621 07:49:42.171136 15546 consensus_queue.cc:260] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P ee318ad5e75847f49a88290b7d43fcde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee318ad5e75847f49a88290b7d43fcde" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43655 } } peers { permanent_uuid: "a0f1558e9a154c5c9f2c171095b2d466" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 36799 } } peers { permanent_uuid: "74936c7336d24fd3b3070e3ff5b2105b" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 36707 } }
I20260621 07:49:42.171670 15534 heartbeater.cc:503] Master 127.0.192.254:42677 was elected leader, sending a full tablet report...
I20260621 07:49:42.171897 15546 ts_tablet_manager.cc:1428] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P ee318ad5e75847f49a88290b7d43fcde: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260621 07:49:42.185406 15400 tablet.cc:2369] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P a0f1558e9a154c5c9f2c171095b2d466: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:49:42.217836 15402 raft_consensus.cc:482] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P a0f1558e9a154c5c9f2c171095b2d466 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:49:42.217967 15402 raft_consensus.cc:504] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P a0f1558e9a154c5c9f2c171095b2d466 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee318ad5e75847f49a88290b7d43fcde" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43655 } } peers { permanent_uuid: "a0f1558e9a154c5c9f2c171095b2d466" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 36799 } } peers { permanent_uuid: "74936c7336d24fd3b3070e3ff5b2105b" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 36707 } }
I20260621 07:49:42.218827 15402 leader_election.cc:288] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P a0f1558e9a154c5c9f2c171095b2d466 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee318ad5e75847f49a88290b7d43fcde (127.0.192.195:43655), 74936c7336d24fd3b3070e3ff5b2105b (127.0.192.193:36707)
I20260621 07:49:42.222586 15401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:42.219072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.195:43655 (local address 127.0.192.194:46375)
0621 07:49:42.219194 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:42.219211 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:49:42.219319 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:49:42.219567 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:49:42.219570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:49:42.219587 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:49:42.219726 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:42.219731 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:42.220476 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:42.220479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:42.221309 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:42.221317 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:42.222108 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:42.222112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:42.222242 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:42.222302 (+ 60us) client_negotiation.cc:769] Sending connection context
0621 07:49:42.222375 (+ 73us) client_negotiation.cc:240] Negotiation successful
0621 07:49:42.222445 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":1}
I20260621 07:49:42.222621 15543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:42.219150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:46375 (local address 127.0.192.195:43655)
0621 07:49:42.219281 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:42.219285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:42.219301 (+ 16us) server_negotiation.cc:407] Connection header received
0621 07:49:42.219431 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:42.219434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:42.219481 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:42.219541 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:49:42.219829 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:42.220379 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:42.221426 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:42.221939 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:42.222085 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:42.222144 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:42.222395 (+ 251us) server_negotiation.cc:299] Negotiation successful
0621 07:49:42.222472 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260621 07:49:42.223125 15267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:42.219421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:45837 (local address 127.0.192.193:36707)
0621 07:49:42.219544 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:42.219549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:42.219664 (+ 115us) server_negotiation.cc:407] Connection header received
0621 07:49:42.219742 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:42.219745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:42.219804 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:42.219881 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:49:42.220118 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:42.220647 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:42.221665 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:42.222220 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:42.222332 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:42.222462 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:42.222932 (+ 470us) server_negotiation.cc:299] Negotiation successful
0621 07:49:42.223027 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 07:49:42.223150 15488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdca2e1b1ce14d44a1868ecc7c4c2e37" candidate_uuid: "a0f1558e9a154c5c9f2c171095b2d466" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee318ad5e75847f49a88290b7d43fcde" is_pre_election: true
I20260621 07:49:42.223434 15488 raft_consensus.cc:2454] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P ee318ad5e75847f49a88290b7d43fcde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0f1558e9a154c5c9f2c171095b2d466 in term 0.
I20260621 07:49:42.223482 15549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:42.219316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:36707 (local address 127.0.192.194:45837)
0621 07:49:42.219579 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:42.219591 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:49:42.219675 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:49:42.219912 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:49:42.219914 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:49:42.219929 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:49:42.220025 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:42.220030 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:42.220753 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:42.220755 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:42.221549 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:42.221559 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:42.222534 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:42.222537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:42.222783 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:42.222820 (+ 37us) client_negotiation.cc:769] Sending connection context
0621 07:49:42.223299 (+ 479us) client_negotiation.cc:240] Negotiation successful
0621 07:49:42.223364 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":20,"thread_start_us":102,"threads_started":1}
I20260621 07:49:42.223804 15286 leader_election.cc:302] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P a0f1558e9a154c5c9f2c171095b2d466 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0f1558e9a154c5c9f2c171095b2d466, ee318ad5e75847f49a88290b7d43fcde; no voters:
I20260621 07:49:42.224056 15402 raft_consensus.cc:2781] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P a0f1558e9a154c5c9f2c171095b2d466 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:49:42.224028 15213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdca2e1b1ce14d44a1868ecc7c4c2e37" candidate_uuid: "a0f1558e9a154c5c9f2c171095b2d466" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74936c7336d24fd3b3070e3ff5b2105b" is_pre_election: true
I20260621 07:49:42.224151 15402 raft_consensus.cc:482] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P a0f1558e9a154c5c9f2c171095b2d466 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:49:42.224196 15402 raft_consensus.cc:3037] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P a0f1558e9a154c5c9f2c171095b2d466 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:49:42.224284 15213 raft_consensus.cc:2454] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0f1558e9a154c5c9f2c171095b2d466 in term 0.
I20260621 07:49:42.225212 15402 raft_consensus.cc:504] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P a0f1558e9a154c5c9f2c171095b2d466 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee318ad5e75847f49a88290b7d43fcde" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43655 } } peers { permanent_uuid: "a0f1558e9a154c5c9f2c171095b2d466" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 36799 } } peers { permanent_uuid: "74936c7336d24fd3b3070e3ff5b2105b" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 36707 } }
I20260621 07:49:42.225478 15402 leader_election.cc:288] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P a0f1558e9a154c5c9f2c171095b2d466 [CANDIDATE]: Term 1 election: Requested vote from peers ee318ad5e75847f49a88290b7d43fcde (127.0.192.195:43655), 74936c7336d24fd3b3070e3ff5b2105b (127.0.192.193:36707)
I20260621 07:49:42.225850 15213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdca2e1b1ce14d44a1868ecc7c4c2e37" candidate_uuid: "a0f1558e9a154c5c9f2c171095b2d466" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74936c7336d24fd3b3070e3ff5b2105b"
I20260621 07:49:42.225946 15488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdca2e1b1ce14d44a1868ecc7c4c2e37" candidate_uuid: "a0f1558e9a154c5c9f2c171095b2d466" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee318ad5e75847f49a88290b7d43fcde"
I20260621 07:49:42.225982 15213 raft_consensus.cc:3037] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:49:42.226068 15488 raft_consensus.cc:3037] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P ee318ad5e75847f49a88290b7d43fcde [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:49:42.227035 15213 raft_consensus.cc:2454] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0f1558e9a154c5c9f2c171095b2d466 in term 1.
I20260621 07:49:42.227069 15488 raft_consensus.cc:2454] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P ee318ad5e75847f49a88290b7d43fcde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0f1558e9a154c5c9f2c171095b2d466 in term 1.
I20260621 07:49:42.227283 15286 leader_election.cc:302] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P a0f1558e9a154c5c9f2c171095b2d466 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74936c7336d24fd3b3070e3ff5b2105b, a0f1558e9a154c5c9f2c171095b2d466; no voters:
I20260621 07:49:42.227458 15402 raft_consensus.cc:2781] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P a0f1558e9a154c5c9f2c171095b2d466 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:49:42.227737 15402 raft_consensus.cc:686] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P a0f1558e9a154c5c9f2c171095b2d466 [term 1 LEADER]: Becoming Leader. State: Replica: a0f1558e9a154c5c9f2c171095b2d466, State: Running, Role: LEADER
I20260621 07:49:42.227967 15402 consensus_queue.cc:237] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P a0f1558e9a154c5c9f2c171095b2d466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee318ad5e75847f49a88290b7d43fcde" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43655 } } peers { permanent_uuid: "a0f1558e9a154c5c9f2c171095b2d466" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 36799 } } peers { permanent_uuid: "74936c7336d24fd3b3070e3ff5b2105b" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 36707 } }
I20260621 07:49:42.230549 15071 catalog_manager.cc:5495] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P a0f1558e9a154c5c9f2c171095b2d466 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0f1558e9a154c5c9f2c171095b2d466 (127.0.192.194). New cstate: current_term: 1 leader_uuid: "a0f1558e9a154c5c9f2c171095b2d466" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee318ad5e75847f49a88290b7d43fcde" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0f1558e9a154c5c9f2c171095b2d466" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 36799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74936c7336d24fd3b3070e3ff5b2105b" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 36707 } health_report { overall_health: UNKNOWN } } }
I20260621 07:49:42.252170 15404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:42.249357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45066 (local address 127.0.192.194:36799)
0621 07:49:42.249478 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:42.249481 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:42.249492 (+ 11us) server_negotiation.cc:407] Connection header received
0621 07:49:42.249583 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:42.249585 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:42.249626 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:42.249701 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:49:42.249865 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:42.250372 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:42.251092 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:42.251315 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:42.251381 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:42.251566 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:49:42.251897 (+ 331us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:49:42.251986 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:42.252055 (+ 69us) server_negotiation.cc:299] Negotiation successful
0621 07:49:42.252090 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 07:49:42.255475 15213 raft_consensus.cc:1261] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [term 1 FOLLOWER]: Refusing update from remote peer a0f1558e9a154c5c9f2c171095b2d466: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:49:42.255510 15488 raft_consensus.cc:1261] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P ee318ad5e75847f49a88290b7d43fcde [term 1 FOLLOWER]: Refusing update from remote peer a0f1558e9a154c5c9f2c171095b2d466: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260621 07:49:42.255728 15260 tablet.cc:2369] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:49:42.256083 15402 consensus_queue.cc:1035] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P a0f1558e9a154c5c9f2c171095b2d466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74936c7336d24fd3b3070e3ff5b2105b" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 36707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:49:42.256234 15550 consensus_queue.cc:1035] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P a0f1558e9a154c5c9f2c171095b2d466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee318ad5e75847f49a88290b7d43fcde" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:49:42.260905 15552 mvcc.cc:205] Tried to move back new op lower bound from 7299187434513637376 to 7299187434408665088. Current Snapshot: MvccSnapshot[applied={T|T < 7299187434513637376}]
I20260621 07:49:42.262640 15553 mvcc.cc:205] Tried to move back new op lower bound from 7299187434513637376 to 7299187434408665088. Current Snapshot: MvccSnapshot[applied={T|T < 7299187434513637376}]
I20260621 07:49:42.263556 15555 mvcc.cc:205] Tried to move back new op lower bound from 7299187434513637376 to 7299187434408665088. Current Snapshot: MvccSnapshot[applied={T|T < 7299187434513637376}]
I20260621 07:49:42.268961 15071 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:41588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:49:42.269094 15071 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:41588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:49:42.271049 15071 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a: Sending DeleteTablet for 3 replicas of tablet cdca2e1b1ce14d44a1868ecc7c4c2e37
I20260621 07:49:42.271569 15332 tablet_service.cc:1514] Processing DeleteTablet for tablet cdca2e1b1ce14d44a1868ecc7c4c2e37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:49:42 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:45054
I20260621 07:49:42.271620 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 15109
I20260621 07:49:42.271629 15193 tablet_service.cc:1514] Processing DeleteTablet for tablet cdca2e1b1ce14d44a1868ecc7c4c2e37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:49:42 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:51832
I20260621 07:49:42.271718 15405 tablet_replica.cc:330] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P a0f1558e9a154c5c9f2c171095b2d466: stopping tablet replica
I20260621 07:49:42.271827 15405 raft_consensus.cc:2229] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P a0f1558e9a154c5c9f2c171095b2d466 [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:49:42.272068 15405 raft_consensus.cc:2258] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P a0f1558e9a154c5c9f2c171095b2d466 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:49:42.272541 15468 tablet_service.cc:1514] Processing DeleteTablet for tablet cdca2e1b1ce14d44a1868ecc7c4c2e37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:49:42 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:58038
I20260621 07:49:42.273747 15561 tablet_replica.cc:330] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P ee318ad5e75847f49a88290b7d43fcde: stopping tablet replica
I20260621 07:49:42.273794 15405 ts_tablet_manager.cc:1905] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P a0f1558e9a154c5c9f2c171095b2d466: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:49:42.273984 15561 raft_consensus.cc:2229] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P ee318ad5e75847f49a88290b7d43fcde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:49:42.274219 15561 raft_consensus.cc:2258] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P ee318ad5e75847f49a88290b7d43fcde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:49:42.276501 15561 ts_tablet_manager.cc:1905] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P ee318ad5e75847f49a88290b7d43fcde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:49:42.276774 15405 ts_tablet_manager.cc:1918] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P a0f1558e9a154c5c9f2c171095b2d466: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:49:42.276846 15405 log.cc:1196] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P a0f1558e9a154c5c9f2c171095b2d466: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/wal/wals/cdca2e1b1ce14d44a1868ecc7c4c2e37
I20260621 07:49:42.277096 15405 ts_tablet_manager.cc:1939] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P a0f1558e9a154c5c9f2c171095b2d466: Deleting consensus metadata
I20260621 07:49:42.277715 15056 catalog_manager.cc:4841] TS a0f1558e9a154c5c9f2c171095b2d466 (127.0.192.194:36799): tablet cdca2e1b1ce14d44a1868ecc7c4c2e37 (table test-table [id=1bd8fa3267654ed2995636a593379dac]) successfully deleted
I20260621 07:49:42.278716 15561 ts_tablet_manager.cc:1918] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P ee318ad5e75847f49a88290b7d43fcde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:49:42.278828 15561 log.cc:1196] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P ee318ad5e75847f49a88290b7d43fcde: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/wal/wals/cdca2e1b1ce14d44a1868ecc7c4c2e37
I20260621 07:49:42.279158 15561 ts_tablet_manager.cc:1939] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P ee318ad5e75847f49a88290b7d43fcde: Deleting consensus metadata
W20260621 07:49:42.279902 15057 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.192.193:36707 (error 108)
I20260621 07:49:42.280295 15057 catalog_manager.cc:4841] TS ee318ad5e75847f49a88290b7d43fcde (127.0.192.195:43655): tablet cdca2e1b1ce14d44a1868ecc7c4c2e37 (table test-table [id=1bd8fa3267654ed2995636a593379dac]) successfully deleted
W20260621 07:49:42.281147 15266 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:49:42.280534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:36707 (local address 127.0.0.1:51838)
0621 07:49:42.280678 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:42.280693 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:49:42.280777 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:49:42.281042 (+ 265us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.193:36707: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":63}
I20260621 07:49:42.281296 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 15264
W20260621 07:49:42.281303 15057 catalog_manager.cc:4584] TS 74936c7336d24fd3b3070e3ff5b2105b (127.0.192.193:36707): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cdca2e1b1ce14d44a1868ecc7c4c2e37: Network error: Client connection negotiation failed: client connection to 127.0.192.193:36707: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260621 07:49:42.286865 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 15403
I20260621 07:49:42.292111 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 15038
I20260621 07:49:42.297868 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:42677
--webserver_interface=127.0.192.254
--webserver_port=41761
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:36019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:42677
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:49:42.396678 15563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:49:42.396934 15563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:49:42.397004 15563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:49:42.400609 15563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:49:42.400686 15563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:49:42.400720 15563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:49:42.400755 15563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:49:42.400842 15563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:49:42.405820 15563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:36019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:42677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:42677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=41761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:49:42.406709 15563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:49:42.407541 15563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:49:42.414674 15568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:49:42.414841 15563 server_base.cc:1034] running on GCE node
W20260621 07:49:42.415081 15569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:49:42.415100 15571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:49:42.415669 15563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:49:42.416657 15563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:49:42.417812 15563 hybrid_clock.cc:648] HybridClock initialized: now 1782028182417800 us; error 22 us; skew 500 ppm
Jun 21 07:49:42 dist-test-slave-lzx3 krb5kdc[14489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:49:42.420099 15563 init.cc:377] Logged in from keytab as kudu/127.0.192.254@KRBTEST.COM (short username kudu)
I20260621 07:49:42.420852 15563 webserver.cc:460] Webserver started at http://127.0.192.254:41761/ using document root <none> and password file <none>
I20260621 07:49:42.421332 15563 fs_manager.cc:359] Metadata directory not provided
I20260621 07:49:42.421430 15563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:49:42.424839 15563 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260621 07:49:42.426752 15578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:42.427714 15563 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:49:42.427834 15563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "814aaaa5c60c4ca6a09cde0649decd1a"
format_stamp: "Formatted at 2026-06-21 07:49:40 on dist-test-slave-lzx3"
server_key: "514c1a28bb9e41ac7f516823c4ee3437"
server_key_iv: "9eddee813aeb52d38b7a1e12ca8afa73"
server_key_version: "encryptionkey@0"
I20260621 07:49:42.428270 15563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:49:42.446945 15563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:49:42.449383 15563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:49:42.449550 15563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:49:42.455828 15563 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:42677
I20260621 07:49:42.456087 15630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:42677 every 8 connection(s)
I20260621 07:49:42.456821 15563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:49:42.460502 15631 sys_catalog.cc:263] Verifying existing consensus state
I20260621 07:49:42.461956 15631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a: Bootstrap starting.
I20260621 07:49:42.463083 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 15563
I20260621 07:49:42.463471 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.193:36707
--local_ip_for_outbound_sockets=127.0.192.193
--tserver_master_addrs=127.0.192.254:42677
--webserver_port=46859
--webserver_interface=127.0.192.193
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:36019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:49:42.477233 15631 log.cc:824] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a: Log is configured to *not* fsync() on all Append() calls
I20260621 07:49:42.490260 15631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 07:49:42.490598 15631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a: Bootstrap complete.
I20260621 07:49:42.495434 15631 raft_consensus.cc:348] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814aaaa5c60c4ca6a09cde0649decd1a" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 42677 } }
I20260621 07:49:42.495975 15631 raft_consensus.cc:729] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 814aaaa5c60c4ca6a09cde0649decd1a, State: Initialized, Role: FOLLOWER
I20260621 07:49:42.496498 15631 consensus_queue.cc:260] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814aaaa5c60c4ca6a09cde0649decd1a" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 42677 } }
I20260621 07:49:42.496670 15631 raft_consensus.cc:388] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:49:42.496738 15631 raft_consensus.cc:482] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:49:42.496829 15631 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [term 2 FOLLOWER]: Advancing to term 3
I20260621 07:49:42.498956 15631 raft_consensus.cc:504] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814aaaa5c60c4ca6a09cde0649decd1a" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 42677 } }
I20260621 07:49:42.499306 15631 leader_election.cc:302] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 814aaaa5c60c4ca6a09cde0649decd1a; no voters:
I20260621 07:49:42.499634 15631 leader_election.cc:288] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [CANDIDATE]: Term 3 election: Requested vote from peers
I20260621 07:49:42.499732 15635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [term 3 FOLLOWER]: Leader election won for term 3
I20260621 07:49:42.499969 15635 raft_consensus.cc:686] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [term 3 LEADER]: Becoming Leader. State: Replica: 814aaaa5c60c4ca6a09cde0649decd1a, State: Running, Role: LEADER
I20260621 07:49:42.500243 15635 consensus_queue.cc:237] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814aaaa5c60c4ca6a09cde0649decd1a" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 42677 } }
I20260621 07:49:42.500847 15631 sys_catalog.cc:564] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:49:42.501150 15635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 814aaaa5c60c4ca6a09cde0649decd1a. Latest consensus state: current_term: 3 leader_uuid: "814aaaa5c60c4ca6a09cde0649decd1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814aaaa5c60c4ca6a09cde0649decd1a" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 42677 } } }
I20260621 07:49:42.501246 15635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [sys.catalog]: This master's current role is: LEADER
I20260621 07:49:42.501538 15639 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:49:42.501916 15636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "814aaaa5c60c4ca6a09cde0649decd1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814aaaa5c60c4ca6a09cde0649decd1a" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 42677 } } }
I20260621 07:49:42.502081 15636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 814aaaa5c60c4ca6a09cde0649decd1a [sys.catalog]: This master's current role is: LEADER
I20260621 07:49:42.506260 15639 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:49:42.506916 15639 catalog_manager.cc:1259] Loaded cluster ID: 138f75245e30495293a83cb39ad5ff64
I20260621 07:49:42.506984 15639 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260621 07:49:42.507787 15639 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f3ba47879ad google::LogMessage::Fail() at ??:0
@ 0x7f3ba478a88a google::LogMessage::SendToLog() at ??:0
@ 0x7f3ba47874de google::LogMessage::Flush() at ??:0
@ 0x7f3ba4788a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3baf0cafdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f3baf0cbd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f3baf0c8579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f3baf10c640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f3ba69d93d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f3ba69d9c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f3ba69db3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f3ba69ca512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f3ba800e6db start_thread at ??:0
@ 0x7f3ba328561f clone at ??:0
W20260621 07:49:42.596388 15633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:49:42.596668 15633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:49:42.596740 15633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:49:42.600517 15633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:49:42.600607 15633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:49:42.600721 15633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.193
I20260621 07:49:42.605661 15633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:36019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.193:36707
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.192.193
--webserver_port=46859
--tserver_master_addrs=127.0.192.254:42677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.193
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:49:42.606642 15633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:49:42.607551 15633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:49:42.617301 15673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:49:42.617374 15675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:49:42.617300 15672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:49:42.617770 15633 server_base.cc:1034] running on GCE node
I20260621 07:49:42.618191 15633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:49:42.618793 15633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:49:42.620002 15633 hybrid_clock.cc:648] HybridClock initialized: now 1782028182619977 us; error 48 us; skew 500 ppm
Jun 21 07:49:42 dist-test-slave-lzx3 krb5kdc[14489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:49:42.622776 15633 init.cc:377] Logged in from keytab as kudu/127.0.192.193@KRBTEST.COM (short username kudu)
I20260621 07:49:42.623713 15633 webserver.cc:460] Webserver started at http://127.0.192.193:46859/ using document root <none> and password file <none>
I20260621 07:49:42.624224 15633 fs_manager.cc:359] Metadata directory not provided
I20260621 07:49:42.624315 15633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:49:42.628108 15633 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:49:42.630396 15682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:42.631604 15633 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 07:49:42.631731 15633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/wal
uuid: "74936c7336d24fd3b3070e3ff5b2105b"
format_stamp: "Formatted at 2026-06-21 07:49:40 on dist-test-slave-lzx3"
server_key: "619b4bc1ac893ba2ce926b0f86126a56"
server_key_iv: "b462ad764d7f3735f8c06fda5a8ac4e0"
server_key_version: "encryptionkey@0"
I20260621 07:49:42.632221 15633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:49:42.642041 15633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:49:42.644716 15633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:49:42.644872 15633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:49:42.645352 15633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:49:42.646339 15689 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260621 07:49:42.651036 15633 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260621 07:49:42.651100 15633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260621 07:49:42.651211 15633 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260621 07:49:42.652652 15633 ts_tablet_manager.cc:610] Registered 1 tablets
I20260621 07:49:42.652705 15633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260621 07:49:42.653002 15689 tablet_bootstrap.cc:492] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b: Bootstrap starting.
I20260621 07:49:42.662366 15633 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.193:36707
I20260621 07:49:42.663184 15633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
I20260621 07:49:42.664072 15796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.193:36707 every 8 connection(s)
I20260621 07:49:42.669438 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 15633
I20260621 07:49:42.669767 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.194:36799
--local_ip_for_outbound_sockets=127.0.192.194
--tserver_master_addrs=127.0.192.254:42677
--webserver_port=33447
--webserver_interface=127.0.192.194
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:36019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:49:42.672646 15689 log.cc:824] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b: Log is configured to *not* fsync() on all Append() calls
I20260621 07:49:42.681510 15689 tablet_bootstrap.cc:492] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 07:49:42.681914 15689 tablet_bootstrap.cc:492] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b: Bootstrap complete.
I20260621 07:49:42.683285 15689 ts_tablet_manager.cc:1397] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b: Time spent bootstrapping tablet: real 0.031s user 0.019s sys 0.008s
I20260621 07:49:42.686245 15689 raft_consensus.cc:348] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee318ad5e75847f49a88290b7d43fcde" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43655 } } peers { permanent_uuid: "a0f1558e9a154c5c9f2c171095b2d466" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 36799 } } peers { permanent_uuid: "74936c7336d24fd3b3070e3ff5b2105b" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 36707 } }
I20260621 07:49:42.686740 15689 raft_consensus.cc:729] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74936c7336d24fd3b3070e3ff5b2105b, State: Initialized, Role: FOLLOWER
I20260621 07:49:42.687207 15689 consensus_queue.cc:260] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee318ad5e75847f49a88290b7d43fcde" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43655 } } peers { permanent_uuid: "a0f1558e9a154c5c9f2c171095b2d466" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 36799 } } peers { permanent_uuid: "74936c7336d24fd3b3070e3ff5b2105b" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 36707 } }
I20260621 07:49:42.688095 15689 ts_tablet_manager.cc:1428] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
W20260621 07:49:42.692770 15799 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:49:42.664913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.193:39521)
0621 07:49:42.665361 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:42.665416 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0621 07:49:42.666569 (+ 1153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:49:42.692066 (+ 25497us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":101,"threads_started":1}
W20260621 07:49:42.693186 15797 heartbeater.cc:644] Failed to heartbeat to 127.0.192.254:42677 (0 consecutive failures): Network error: Failed to ping master at 127.0.192.254:42677: Client connection negotiation failed: client connection to 127.0.192.254:42677: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260621 07:49:42.693801 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:42.693552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.193:47827)
0621 07:49:42.693662 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:42.693740 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260621 07:49:42.694368 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:42.694186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.193:48091)
0621 07:49:42.694283 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:42.694329 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260621 07:49:42.694504 15797 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 07:49:42.775830 15800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:49:42.776077 15800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:49:42.776146 15800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:49:42.779790 15800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:49:42.779892 15800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:49:42.779973 15800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.194
I20260621 07:49:42.784507 15800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:36019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.194:36799
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.192.194
--webserver_port=33447
--tserver_master_addrs=127.0.192.254:42677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.194
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:49:42.785410 15800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:49:42.786322 15800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:49:42.794464 15810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:49:42.794477 15808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:49:42.794477 15807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:49:42.794621 15800 server_base.cc:1034] running on GCE node
I20260621 07:49:42.795109 15800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:49:42.795687 15800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:49:42.796866 15800 hybrid_clock.cc:648] HybridClock initialized: now 1782028182796853 us; error 32 us; skew 500 ppm
Jun 21 07:49:42 dist-test-slave-lzx3 krb5kdc[14489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:49:42.799252 15800 init.cc:377] Logged in from keytab as kudu/127.0.192.194@KRBTEST.COM (short username kudu)
I20260621 07:49:42.799990 15800 webserver.cc:460] Webserver started at http://127.0.192.194:33447/ using document root <none> and password file <none>
I20260621 07:49:42.800499 15800 fs_manager.cc:359] Metadata directory not provided
I20260621 07:49:42.800606 15800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:49:42.803881 15800 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:49:42.805751 15817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:42.806948 15800 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 07:49:42.807071 15800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/wal
uuid: "a0f1558e9a154c5c9f2c171095b2d466"
format_stamp: "Formatted at 2026-06-21 07:49:40 on dist-test-slave-lzx3"
server_key: "d9b79482b0666d353e0c86f31c4c0f49"
server_key_iv: "d4705c943e09bad2d1a0d6837cb9abd2"
server_key_version: "encryptionkey@0"
I20260621 07:49:42.807710 15800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:49:42.823725 15800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:49:42.826280 15800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:49:42.826495 15800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:49:42.827013 15800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:49:42.827751 15800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:49:42.827790 15800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:42.827845 15800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:49:42.827874 15800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:42.835569 15800 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.194:36799
I20260621 07:49:42.835597 15930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.194:36799 every 8 connection(s)
I20260621 07:49:42.836483 15800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
I20260621 07:49:42.839576 15933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:42.838288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.194:37391)
0621 07:49:42.838898 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:42.838987 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":106,"threads_started":1}
W20260621 07:49:42.840059 15931 heartbeater.cc:644] Failed to heartbeat to 127.0.192.254:42677 (0 consecutive failures): Network error: Failed to ping master at 127.0.192.254:42677: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
I20260621 07:49:42.840597 15933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:42.840399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.194:57867)
0621 07:49:42.840496 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:42.840535 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260621 07:49:42.841133 15933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:42.840970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.194:39247)
0621 07:49:42.841055 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:42.841091 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260621 07:49:42.841305 15931 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260621 07:49:42.846145 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 15800
I20260621 07:49:42.846546 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.195:43655
--local_ip_for_outbound_sockets=127.0.192.195
--tserver_master_addrs=127.0.192.254:42677
--webserver_port=38937
--webserver_interface=127.0.192.195
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:36019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:49:42.942406 15934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:49:42.942600 15934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:49:42.942662 15934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:49:42.946167 15934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:49:42.946211 15934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:49:42.946305 15934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.195
I20260621 07:49:42.950929 15934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:36019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.195:43655
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.192.195
--webserver_port=38937
--tserver_master_addrs=127.0.192.254:42677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.195
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:49:42.951651 15934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:49:42.952414 15934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:49:42.960307 15939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:49:42.960388 15940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:49:42.960314 15942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:49:42.960631 15934 server_base.cc:1034] running on GCE node
I20260621 07:49:42.961063 15934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:49:42.961620 15934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:49:42.962814 15934 hybrid_clock.cc:648] HybridClock initialized: now 1782028182962785 us; error 48 us; skew 500 ppm
Jun 21 07:49:42 dist-test-slave-lzx3 krb5kdc[14489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:49:42.965229 15934 init.cc:377] Logged in from keytab as kudu/127.0.192.195@KRBTEST.COM (short username kudu)
I20260621 07:49:42.965972 15934 webserver.cc:460] Webserver started at http://127.0.192.195:38937/ using document root <none> and password file <none>
I20260621 07:49:42.966424 15934 fs_manager.cc:359] Metadata directory not provided
I20260621 07:49:42.966475 15934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:49:42.969458 15934 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260621 07:49:42.971343 15949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:42.972458 15934 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:49:42.972543 15934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/wal
uuid: "ee318ad5e75847f49a88290b7d43fcde"
format_stamp: "Formatted at 2026-06-21 07:49:40 on dist-test-slave-lzx3"
server_key: "7f7ea3f47e3b338096e6055e96534c32"
server_key_iv: "e68ad8c496595a39d94c6e5694352ec9"
server_key_version: "encryptionkey@0"
I20260621 07:49:42.973059 15934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:49:42.993371 15934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:49:42.996353 15934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:49:42.996539 15934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:49:42.997076 15934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:49:42.997781 15934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:49:42.997833 15934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:42.997896 15934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:49:42.997936 15934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:43.006258 15934 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.195:43655
I20260621 07:49:43.006299 16062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.195:43655 every 8 connection(s)
I20260621 07:49:43.007134 15934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
I20260621 07:49:43.010360 16065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:43.009072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.195:49601)
0621 07:49:43.009646 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:43.009740 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":102,"threads_started":1}
W20260621 07:49:43.010854 16063 heartbeater.cc:644] Failed to heartbeat to 127.0.192.254:42677 (0 consecutive failures): Network error: Failed to ping master at 127.0.192.254:42677: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
I20260621 07:49:43.011420 16065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:43.011182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.195:36027)
0621 07:49:43.011306 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:43.011372 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260621 07:49:43.011817 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 15934
I20260621 07:49:43.012214 16065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:43.011991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.195:50325)
0621 07:49:43.012100 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:43.012148 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260621 07:49:43.012387 16063 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260621 07:49:43.695523 16067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:43.695049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.193:57985)
0621 07:49:43.695327 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:43.695396 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":78,"threads_started":1}
I20260621 07:49:43.842105 16068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:43.841761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.194:48601)
0621 07:49:43.841972 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:43.842019 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":65,"threads_started":1}
I20260621 07:49:43.968011 16069 raft_consensus.cc:482] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:49:43.968191 16069 raft_consensus.cc:504] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee318ad5e75847f49a88290b7d43fcde" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43655 } } peers { permanent_uuid: "a0f1558e9a154c5c9f2c171095b2d466" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 36799 } } peers { permanent_uuid: "74936c7336d24fd3b3070e3ff5b2105b" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 36707 } }
I20260621 07:49:43.969306 16069 leader_election.cc:288] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee318ad5e75847f49a88290b7d43fcde (127.0.192.195:43655), a0f1558e9a154c5c9f2c171095b2d466 (127.0.192.194:36799)
Jun 21 07:49:43 dist-test-slave-lzx3 krb5kdc[14489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for kudu/127.0.192.194@KRBTEST.COM
Jun 21 07:49:43 dist-test-slave-lzx3 krb5kdc[14489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for kudu/127.0.192.195@KRBTEST.COM
I20260621 07:49:43.983242 16067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:43.969556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:36799 (local address 127.0.192.193:50249)
0621 07:49:43.969685 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:43.969699 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:49:43.969805 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:49:43.971748 (+ 1943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:49:43.971753 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:49:43.972096 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:49:43.972527 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:43.972540 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:43.973856 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:43.973859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:43.974760 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:43.974769 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:43.975449 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:43.975453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:43.975655 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:43.976349 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:49:43.976364 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:49:43.977931 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:49:43.980331 (+ 2400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:43.980342 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:43.980350 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:43.980815 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:43.981241 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:43.981245 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:43.981246 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:43.981425 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:43.982177 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:49:43.982181 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:49:43.982507 (+ 326us) client_negotiation.cc:769] Sending connection context
0621 07:49:43.982695 (+ 188us) client_negotiation.cc:240] Negotiation successful
0621 07:49:43.983087 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":141,"spinlock_wait_cycles":40960}
I20260621 07:49:43.983278 16071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:43.969738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:50249 (local address 127.0.192.194:36799)
0621 07:49:43.970050 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:43.970074 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:43.970114 (+ 40us) server_negotiation.cc:407] Connection header received
0621 07:49:43.970967 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:43.970973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:43.971385 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:43.971702 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:49:43.972783 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:43.973706 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:43.974886 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:43.975327 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:43.975498 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:43.978173 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:49:43.978178 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:49:43.978188 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:49:43.978212 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:49:43.980193 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:43.980945 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:43.980950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:43.980956 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:43.981141 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:43.981557 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:43.981563 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:43.981565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:43.982065 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:49:43.982205 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:43.982843 (+ 638us) server_negotiation.cc:299] Negotiation successful
0621 07:49:43.983088 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":82,"threads_started":1}
I20260621 07:49:43.983281 16072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:43.969763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:43427 (local address 127.0.192.195:43655)
0621 07:49:43.970150 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:43.970159 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:43.970187 (+ 28us) server_negotiation.cc:407] Connection header received
0621 07:49:43.970950 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:43.970959 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:43.971385 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:43.971702 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:49:43.972764 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:43.973716 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:43.974886 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:43.975336 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:43.975515 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:43.978392 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:49:43.978397 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:49:43.978406 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:49:43.978428 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:49:43.980193 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:43.980946 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:43.980950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:43.980956 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:43.981141 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:43.981727 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:43.981730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:43.981732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:43.982136 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:49:43.982258 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:43.982839 (+ 581us) server_negotiation.cc:299] Negotiation successful
0621 07:49:43.983088 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":107,"threads_started":1}
I20260621 07:49:43.983243 16070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:43.969556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.195:43655 (local address 127.0.192.193:43427)
0621 07:49:43.970063 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:43.970075 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:49:43.970151 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:49:43.972154 (+ 2003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:49:43.972156 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:49:43.972295 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:49:43.972526 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:43.972540 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:43.973864 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:43.973871 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:43.974762 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:43.974770 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:43.975457 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:43.975461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:43.975676 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:43.976394 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:49:43.976405 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:49:43.978226 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:49:43.980331 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:43.980336 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:43.980350 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:43.980815 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:43.981546 (+ 731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:43.981549 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:43.981551 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:43.981627 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:43.982241 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:49:43.982243 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:49:43.982507 (+ 264us) client_negotiation.cc:769] Sending connection context
0621 07:49:43.982695 (+ 188us) client_negotiation.cc:240] Negotiation successful
0621 07:49:43.983041 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":337,"spinlock_wait_cycles":17280,"thread_start_us":82,"threads_started":1}
W20260621 07:49:43.986142 15685 leader_election.cc:341] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ee318ad5e75847f49a88290b7d43fcde (127.0.192.195:43655): Not found: Tablet not found: cdca2e1b1ce14d44a1868ecc7c4c2e37
W20260621 07:49:43.986331 15684 leader_election.cc:341] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a0f1558e9a154c5c9f2c171095b2d466 (127.0.192.194:36799): Not found: Tablet not found: cdca2e1b1ce14d44a1868ecc7c4c2e37
I20260621 07:49:43.986455 15684 leader_election.cc:302] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74936c7336d24fd3b3070e3ff5b2105b; no voters: a0f1558e9a154c5c9f2c171095b2d466, ee318ad5e75847f49a88290b7d43fcde
I20260621 07:49:43.986682 16069 raft_consensus.cc:2726] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 07:49:44.013244 16073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:44.012847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.195:39505)
0621 07:49:44.013059 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:44.013123 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":89,"threads_started":1}
I20260621 07:49:44.696692 16074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:44.696162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.193:46281)
0621 07:49:44.696501 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:44.696606 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":62,"threads_started":1}
I20260621 07:49:44.843024 16075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:44.842708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.194:36985)
0621 07:49:44.842910 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:44.842954 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":62,"threads_started":1}
I20260621 07:49:45.014232 16076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:45.013864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.195:45425)
0621 07:49:45.014098 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:45.014148 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":68,"threads_started":1}
I20260621 07:49:45.697685 16077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:45.697321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.193:48685)
0621 07:49:45.697547 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:45.697598 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":73,"threads_started":1}
I20260621 07:49:45.831626 16078 raft_consensus.cc:482] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:49:45.831746 16078 raft_consensus.cc:504] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee318ad5e75847f49a88290b7d43fcde" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43655 } } peers { permanent_uuid: "a0f1558e9a154c5c9f2c171095b2d466" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 36799 } } peers { permanent_uuid: "74936c7336d24fd3b3070e3ff5b2105b" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 36707 } }
I20260621 07:49:45.832099 16078 leader_election.cc:288] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee318ad5e75847f49a88290b7d43fcde (127.0.192.195:43655), a0f1558e9a154c5c9f2c171095b2d466 (127.0.192.194:36799)
W20260621 07:49:45.832878 15684 leader_election.cc:341] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a0f1558e9a154c5c9f2c171095b2d466 (127.0.192.194:36799): Not found: Tablet not found: cdca2e1b1ce14d44a1868ecc7c4c2e37
W20260621 07:49:45.833007 15685 leader_election.cc:341] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ee318ad5e75847f49a88290b7d43fcde (127.0.192.195:43655): Not found: Tablet not found: cdca2e1b1ce14d44a1868ecc7c4c2e37
I20260621 07:49:45.833112 15685 leader_election.cc:302] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74936c7336d24fd3b3070e3ff5b2105b; no voters: a0f1558e9a154c5c9f2c171095b2d466, ee318ad5e75847f49a88290b7d43fcde
I20260621 07:49:45.833236 16078 raft_consensus.cc:2726] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 07:49:45.843933 16079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:45.843596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.194:57945)
0621 07:49:45.843813 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:45.843861 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":74,"threads_started":1}
I20260621 07:49:46.015298 16080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:46.014902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.195:34215)
0621 07:49:46.015162 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:46.015212 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":80,"threads_started":1}
I20260621 07:49:46.698721 16081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:46.698364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.193:50311)
0621 07:49:46.698593 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:46.698642 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":65,"threads_started":1}
I20260621 07:49:46.845049 16082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:46.844665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.194:39623)
0621 07:49:46.844908 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:46.844965 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":71,"threads_started":1}
I20260621 07:49:47.016302 16083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:47.015936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.195:33715)
0621 07:49:47.016163 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:47.016217 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":78,"threads_started":1}
I20260621 07:49:47.669269 16084 raft_consensus.cc:482] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:49:47.669437 16084 raft_consensus.cc:504] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee318ad5e75847f49a88290b7d43fcde" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43655 } } peers { permanent_uuid: "a0f1558e9a154c5c9f2c171095b2d466" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 36799 } } peers { permanent_uuid: "74936c7336d24fd3b3070e3ff5b2105b" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 36707 } }
I20260621 07:49:47.669862 16084 leader_election.cc:288] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee318ad5e75847f49a88290b7d43fcde (127.0.192.195:43655), a0f1558e9a154c5c9f2c171095b2d466 (127.0.192.194:36799)
W20260621 07:49:47.670564 15685 leader_election.cc:341] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ee318ad5e75847f49a88290b7d43fcde (127.0.192.195:43655): Not found: Tablet not found: cdca2e1b1ce14d44a1868ecc7c4c2e37
W20260621 07:49:47.670651 15684 leader_election.cc:341] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a0f1558e9a154c5c9f2c171095b2d466 (127.0.192.194:36799): Not found: Tablet not found: cdca2e1b1ce14d44a1868ecc7c4c2e37
I20260621 07:49:47.670711 15684 leader_election.cc:302] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74936c7336d24fd3b3070e3ff5b2105b; no voters: a0f1558e9a154c5c9f2c171095b2d466, ee318ad5e75847f49a88290b7d43fcde
I20260621 07:49:47.670867 16084 raft_consensus.cc:2726] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 07:49:47.699980 16085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:47.699555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.193:54087)
0621 07:49:47.699829 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:47.699882 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":68,"threads_started":1}
I20260621 07:49:47.846060 16086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:47.845727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.194:56597)
0621 07:49:47.845934 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:47.845982 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":67,"threads_started":1}
I20260621 07:49:48.017421 16087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:48.017052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.195:58585)
0621 07:49:48.017293 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:48.017344 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":76,"threads_started":1}
I20260621 07:49:48.701066 16088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:48.700680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.193:54455)
0621 07:49:48.700906 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:48.700959 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":57,"threads_started":1}
I20260621 07:49:48.847069 16089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:48.846740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.194:49807)
0621 07:49:48.846948 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:48.846995 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":62,"threads_started":1}
I20260621 07:49:49.018503 16090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:49.018145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.195:48597)
0621 07:49:49.018372 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:49.018421 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":70,"threads_started":1}
I20260621 07:49:49.702126 16091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:49.701746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.193:37521)
0621 07:49:49.701989 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:49.702038 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":89,"threads_started":1}
I20260621 07:49:49.848112 16092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:49.847694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.194:38309)
0621 07:49:49.847981 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:49.848033 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":68,"threads_started":1}
I20260621 07:49:50.019667 16093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:50.019218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.195:56701)
0621 07:49:50.019500 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:50.019568 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":73,"threads_started":1}
I20260621 07:49:50.337440 16094 raft_consensus.cc:482] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:49:50.337570 16094 raft_consensus.cc:504] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee318ad5e75847f49a88290b7d43fcde" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43655 } } peers { permanent_uuid: "a0f1558e9a154c5c9f2c171095b2d466" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 36799 } } peers { permanent_uuid: "74936c7336d24fd3b3070e3ff5b2105b" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 36707 } }
I20260621 07:49:50.338063 16094 leader_election.cc:288] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee318ad5e75847f49a88290b7d43fcde (127.0.192.195:43655), a0f1558e9a154c5c9f2c171095b2d466 (127.0.192.194:36799)
W20260621 07:49:50.338846 15684 leader_election.cc:341] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a0f1558e9a154c5c9f2c171095b2d466 (127.0.192.194:36799): Not found: Tablet not found: cdca2e1b1ce14d44a1868ecc7c4c2e37
W20260621 07:49:50.338956 15685 leader_election.cc:341] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ee318ad5e75847f49a88290b7d43fcde (127.0.192.195:43655): Not found: Tablet not found: cdca2e1b1ce14d44a1868ecc7c4c2e37
I20260621 07:49:50.339022 15685 leader_election.cc:302] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74936c7336d24fd3b3070e3ff5b2105b; no voters: a0f1558e9a154c5c9f2c171095b2d466, ee318ad5e75847f49a88290b7d43fcde
I20260621 07:49:50.339221 16094 raft_consensus.cc:2726] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 07:49:50.703156 16095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:50.702786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.193:51747)
0621 07:49:50.703015 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:50.703066 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":70,"threads_started":1}
I20260621 07:49:50.849159 16096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:50.848798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.194:53575)
0621 07:49:50.849022 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:50.849074 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":63,"threads_started":1}
I20260621 07:49:51.020790 16097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:51.020342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.195:52825)
0621 07:49:51.020650 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:51.020705 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":71,"threads_started":1}
I20260621 07:49:51.704212 16098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:51.703854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.193:52911)
0621 07:49:51.704074 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:51.704123 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":70,"threads_started":1}
I20260621 07:49:51.850258 16099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:51.849781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.194:55191)
0621 07:49:51.850035 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:51.850097 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":81,"threads_started":1}
I20260621 07:49:52.021824 16100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:52.021443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.195:40857)
0621 07:49:52.021686 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:52.021737 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":76,"threads_started":1}
I20260621 07:49:52.705472 16101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:52.705018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.193:47787)
0621 07:49:52.705324 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:52.705380 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":90,"threads_started":1}
I20260621 07:49:52.851432 16102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:52.851025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.194:41761)
0621 07:49:52.851287 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:52.851339 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":68,"threads_started":1}
I20260621 07:49:53.022862 16103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:53.022504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.195:57925)
0621 07:49:53.022730 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:53.022781 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":65,"threads_started":1}
I20260621 07:49:53.706529 16104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:53.706142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.193:53759)
0621 07:49:53.706383 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:53.706437 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":68,"threads_started":1}
I20260621 07:49:53.852495 16105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:53.852098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.194:51967)
0621 07:49:53.852336 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:53.852394 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":75,"threads_started":1}
I20260621 07:49:54.023985 16106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:54.023611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.195:39841)
0621 07:49:54.023837 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:54.023888 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":70,"threads_started":1}
I20260621 07:49:54.439992 16107 raft_consensus.cc:482] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:49:54.440110 16107 raft_consensus.cc:504] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee318ad5e75847f49a88290b7d43fcde" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 43655 } } peers { permanent_uuid: "a0f1558e9a154c5c9f2c171095b2d466" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 36799 } } peers { permanent_uuid: "74936c7336d24fd3b3070e3ff5b2105b" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 36707 } }
I20260621 07:49:54.440600 16107 leader_election.cc:288] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee318ad5e75847f49a88290b7d43fcde (127.0.192.195:43655), a0f1558e9a154c5c9f2c171095b2d466 (127.0.192.194:36799)
W20260621 07:49:54.441298 15685 leader_election.cc:341] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ee318ad5e75847f49a88290b7d43fcde (127.0.192.195:43655): Not found: Tablet not found: cdca2e1b1ce14d44a1868ecc7c4c2e37
W20260621 07:49:54.441399 15684 leader_election.cc:341] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a0f1558e9a154c5c9f2c171095b2d466 (127.0.192.194:36799): Not found: Tablet not found: cdca2e1b1ce14d44a1868ecc7c4c2e37
I20260621 07:49:54.441488 15684 leader_election.cc:302] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74936c7336d24fd3b3070e3ff5b2105b; no voters: a0f1558e9a154c5c9f2c171095b2d466, ee318ad5e75847f49a88290b7d43fcde
I20260621 07:49:54.441627 16107 raft_consensus.cc:2726] T cdca2e1b1ce14d44a1868ecc7c4c2e37 P 74936c7336d24fd3b3070e3ff5b2105b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 07:49:54.707737 16108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:54.707253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.193:59199)
0621 07:49:54.707568 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:54.707620 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":104,"threads_started":1}
I20260621 07:49:54.853613 16109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:54.853209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.194:55153)
0621 07:49:54.853471 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:54.853527 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":81,"threads_started":1}
I20260621 07:49:55.025063 16110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:55.024696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.195:46593)
0621 07:49:55.024925 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:55.024978 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":65,"threads_started":1}
I20260621 07:49:55.708946 16111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:55.708469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.193:56957)
0621 07:49:55.708773 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:55.708848 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":136,"threads_started":1}
I20260621 07:49:55.854748 16112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:55.854325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.194:39387)
0621 07:49:55.854593 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:55.854652 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":88,"threads_started":1}
I20260621 07:49:56.026238 16113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:56.025795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.195:42565)
0621 07:49:56.026078 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:56.026141 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":85,"threads_started":1}
I20260621 07:49:56.710081 16114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:56.709680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.193:44119)
0621 07:49:56.709919 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:56.709974 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":72,"threads_started":1}
I20260621 07:49:56.855949 16115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:56.855518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.194:59669)
0621 07:49:56.855785 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:56.855841 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":112,"threads_started":1}
I20260621 07:49:57.027375 16116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:57.026979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.195:45893)
0621 07:49:57.027219 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:57.027270 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":70,"threads_started":1}
I20260621 07:49:57.711594 16117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:57.711077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.193:53389)
0621 07:49:57.711380 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:57.711478 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":91,"threads_started":1}
I20260621 07:49:57.857189 16118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:57.856816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.194:48347)
0621 07:49:57.857037 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:57.857089 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":69,"threads_started":1}
I20260621 07:49:58.020959 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 15633
I20260621 07:49:58.026595 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 15800
I20260621 07:49:58.028707 16119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:58.028284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:42677 (local address 127.0.192.195:35999)
0621 07:49:58.028522 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:58.028622 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":87,"threads_started":1}
I20260621 07:49:58.031957 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 15934
2026-06-21T07:49:58Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19074 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:49:58 dist-test-slave-lzx3 krb5kdc[16122](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:49:58 dist-test-slave-lzx3 krb5kdc[16122](info): set up 2 sockets
Jun 21 07:49:58 dist-test-slave-lzx3 krb5kdc[16122](info): commencing operation
krb5kdc: starting...
W20260621 07:49:59.064752 771 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/test-user.keytab.
Jun 21 07:49:59 dist-test-slave-lzx3 krb5kdc[16122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:49:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:49:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.192.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
I20260621 07:49:59.179482 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:45149
--webserver_interface=127.0.192.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:35349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:45149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:49:59.286188 16135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:49:59.286509 16135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:49:59.286585 16135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:49:59.290460 16135 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260621 07:49:59.290508 16135 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260621 07:49:59.290526 16135 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260621 07:49:59.290571 16135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:49:59.290611 16135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:49:59.290659 16135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:49:59.290700 16135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:49:59.290753 16135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:49:59.296106 16135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:35349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:45149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:45149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:49:59.297128 16135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:49:59.298118 16135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:49:59.306387 16143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:49:59.306552 16135 server_base.cc:1034] running on GCE node
W20260621 07:49:59.306452 16140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:49:59.306401 16141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:49:59.307298 16135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:49:59.308343 16135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:49:59.309520 16135 hybrid_clock.cc:648] HybridClock initialized: now 1782028199309482 us; error 50 us; skew 500 ppm
Jun 21 07:49:59 dist-test-slave-lzx3 krb5kdc[16122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:49:59.312321 16135 init.cc:377] Logged in from keytab as kudu/127.0.192.254@KRBTEST.COM (short username kudu)
I20260621 07:49:59.313206 16135 webserver.cc:460] Webserver started at http://127.0.192.254:40327/ using document root <none> and password file <none>
I20260621 07:49:59.313764 16135 fs_manager.cc:359] Metadata directory not provided
I20260621 07:49:59.313854 16135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:49:59.314080 16135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:49:59.316095 16135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "98c78c73eb2247848e49787843c20f5c"
format_stamp: "Formatted at 2026-06-21 07:49:59 on dist-test-slave-lzx3"
server_key: "7f1307aaa489feea232904a5e07855ce"
server_key_iv: "a66aad73380eec7c6c8f0531051c12c0"
server_key_version: "encryptionkey@0"
I20260621 07:49:59.316707 16135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "98c78c73eb2247848e49787843c20f5c"
format_stamp: "Formatted at 2026-06-21 07:49:59 on dist-test-slave-lzx3"
server_key: "7f1307aaa489feea232904a5e07855ce"
server_key_iv: "a66aad73380eec7c6c8f0531051c12c0"
server_key_version: "encryptionkey@0"
I20260621 07:49:59.320144 16135 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:49:59.322551 16150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:59.323680 16135 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:49:59.323815 16135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "98c78c73eb2247848e49787843c20f5c"
format_stamp: "Formatted at 2026-06-21 07:49:59 on dist-test-slave-lzx3"
server_key: "7f1307aaa489feea232904a5e07855ce"
server_key_iv: "a66aad73380eec7c6c8f0531051c12c0"
server_key_version: "encryptionkey@0"
I20260621 07:49:59.323928 16135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:49:59.341131 16135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:49:59.343766 16135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:49:59.343973 16135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:49:59.350665 16135 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:45149
I20260621 07:49:59.350942 16202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:45149 every 8 connection(s)
I20260621 07:49:59.351567 16135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:49:59.354235 16203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:49:59.354946 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 16135
I20260621 07:49:59.355082 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:49:59.355361 771 external_mini_cluster.cc:1428] Setting key 55392d808ea3d4c009032e8fca527fe4
I20260621 07:49:59.360204 16203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c: Bootstrap starting.
I20260621 07:49:59.362756 16203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c: Neither blocks nor log segments found. Creating new log.
I20260621 07:49:59.363704 16203 log.cc:824] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c: Log is configured to *not* fsync() on all Append() calls
Jun 21 07:49:59 dist-test-slave-lzx3 krb5kdc[16122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:49:59.365587 16203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c: No bootstrap required, opened a new log
I20260621 07:49:59.369061 16206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:59.356642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50550 (local address 127.0.192.254:45149)
0621 07:49:59.357011 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:59.357025 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:59.357057 (+ 32us) server_negotiation.cc:407] Connection header received
0621 07:49:59.357671 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:59.357687 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:59.357988 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:59.358259 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:49:59.359144 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:59.360108 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:59.361244 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:59.361710 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:59.361828 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:59.364695 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:49:59.364700 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:49:59.364712 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:49:59.364748 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:49:59.366329 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:59.366809 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:59.366813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:59.366817 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:59.366917 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:59.367183 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:59.367185 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:59.367186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:59.367526 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:49:59.367658 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:59.368022 (+ 364us) server_negotiation.cc:299] Negotiation successful
0621 07:49:59.368296 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":118,"threads_started":1}
I20260621 07:49:59.369629 16203 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c78c73eb2247848e49787843c20f5c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45149 } }
I20260621 07:49:59.369885 16203 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:49:59.369932 16203 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98c78c73eb2247848e49787843c20f5c, State: Initialized, Role: FOLLOWER
I20260621 07:49:59.370369 16203 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c78c73eb2247848e49787843c20f5c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45149 } }
I20260621 07:49:59.370522 16203 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:49:59.370596 16203 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:49:59.370718 16203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:49:59.371759 16203 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c78c73eb2247848e49787843c20f5c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45149 } }
I20260621 07:49:59.372082 16203 leader_election.cc:302] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98c78c73eb2247848e49787843c20f5c; no voters:
I20260621 07:49:59.372377 16203 leader_election.cc:288] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:49:59.372545 16208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:49:59.372891 16208 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [term 1 LEADER]: Becoming Leader. State: Replica: 98c78c73eb2247848e49787843c20f5c, State: Running, Role: LEADER
I20260621 07:49:59.373234 16208 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c78c73eb2247848e49787843c20f5c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45149 } }
I20260621 07:49:59.373591 16203 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:49:59.374903 16209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98c78c73eb2247848e49787843c20f5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c78c73eb2247848e49787843c20f5c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45149 } } }
I20260621 07:49:59.374894 16210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98c78c73eb2247848e49787843c20f5c. Latest consensus state: current_term: 1 leader_uuid: "98c78c73eb2247848e49787843c20f5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c78c73eb2247848e49787843c20f5c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45149 } } }
I20260621 07:49:59.375026 16210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [sys.catalog]: This master's current role is: LEADER
I20260621 07:49:59.375026 16209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [sys.catalog]: This master's current role is: LEADER
I20260621 07:49:59.375595 16218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:49:59.378641 16218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:49:59.383745 16218 catalog_manager.cc:1347] Generated new cluster ID: 6831487686084c91b5de07c168ebca26
I20260621 07:49:59.383807 16218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:49:59.410211 16218 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:49:59.411208 16218 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:49:59.418362 16218 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c: Generated new TSK 0
I20260621 07:49:59.419013 16218 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.192.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab.
Entry for principal kudu/127.0.192.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab.
I20260621 07:49:59.449738 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.193:0
--local_ip_for_outbound_sockets=127.0.192.193
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:45149
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:35349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:49:59.569715 16229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:49:59.569952 16229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:49:59.569993 16229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:49:59.574035 16229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:49:59.574088 16229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:49:59.574144 16229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.193
I20260621 07:49:59.579344 16229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:35349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:45149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.193
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:49:59.580232 16229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:49:59.581185 16229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:49:59.590420 16235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:49:59.590463 16229 server_base.cc:1034] running on GCE node
W20260621 07:49:59.590430 16234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:49:59.590430 16237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:49:59.591014 16229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:49:59.591670 16229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:49:59.592850 16229 hybrid_clock.cc:648] HybridClock initialized: now 1782028199592837 us; error 26 us; skew 500 ppm
Jun 21 07:49:59 dist-test-slave-lzx3 krb5kdc[16122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:49:59.595665 16229 init.cc:377] Logged in from keytab as kudu/127.0.192.193@KRBTEST.COM (short username kudu)
I20260621 07:49:59.596540 16229 webserver.cc:460] Webserver started at http://127.0.192.193:42439/ using document root <none> and password file <none>
I20260621 07:49:59.597112 16229 fs_manager.cc:359] Metadata directory not provided
I20260621 07:49:59.597169 16229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:49:59.597327 16229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:49:59.599246 16229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/data/instance:
uuid: "854c12bbec8949f397347e2dc7604940"
format_stamp: "Formatted at 2026-06-21 07:49:59 on dist-test-slave-lzx3"
server_key: "c7258f611355283400ad1fb45aee31a1"
server_key_iv: "0585450526b4f4e32e317a48b8232c9c"
server_key_version: "encryptionkey@0"
I20260621 07:49:59.599738 16229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/wal/instance:
uuid: "854c12bbec8949f397347e2dc7604940"
format_stamp: "Formatted at 2026-06-21 07:49:59 on dist-test-slave-lzx3"
server_key: "c7258f611355283400ad1fb45aee31a1"
server_key_iv: "0585450526b4f4e32e317a48b8232c9c"
server_key_version: "encryptionkey@0"
I20260621 07:49:59.603245 16229 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260621 07:49:59.605460 16244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:59.606607 16229 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260621 07:49:59.606745 16229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/wal
uuid: "854c12bbec8949f397347e2dc7604940"
format_stamp: "Formatted at 2026-06-21 07:49:59 on dist-test-slave-lzx3"
server_key: "c7258f611355283400ad1fb45aee31a1"
server_key_iv: "0585450526b4f4e32e317a48b8232c9c"
server_key_version: "encryptionkey@0"
I20260621 07:49:59.606850 16229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:49:59.632122 16229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:49:59.634840 16229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:49:59.635032 16229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:49:59.635591 16229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:49:59.636353 16229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:49:59.636422 16229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:59.636488 16229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:49:59.636538 16229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:59.644913 16229 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.193:46373
I20260621 07:49:59.644940 16357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.193:46373 every 8 connection(s)
I20260621 07:49:59.645710 16229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
I20260621 07:49:59.645908 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 16229
I20260621 07:49:59.646036 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/wal/instance
I20260621 07:49:59.646346 771 external_mini_cluster.cc:1428] Setting key ed0fa54b397f021e2a87359e70c41b8b
WARNING: no policy specified for kudu/127.0.192.194@KRBTEST.COM; defaulting to no policy
Jun 21 07:49:59 dist-test-slave-lzx3 krb5kdc[16122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.194@KRBTEST.COM" created.
I20260621 07:49:59.661365 16206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:59.647454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:36245 (local address 127.0.192.254:45149)
0621 07:49:59.647579 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:59.647582 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:59.648593 (+ 1011us) server_negotiation.cc:407] Connection header received
0621 07:49:59.650230 (+ 1637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:59.650233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:59.650287 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:59.650364 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:49:59.651842 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:59.652455 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:59.654346 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:59.654574 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:59.654652 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:59.657566 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:49:59.657569 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:49:59.657571 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:49:59.657598 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:49:59.659131 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:59.659731 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:59.659734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:59.659736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:59.659832 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:59.660151 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:59.660157 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:59.660162 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:59.660414 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:49:59.660519 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:59.661056 (+ 537us) server_negotiation.cc:299] Negotiation successful
0621 07:49:59.661182 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260621 07:49:59.662194 16360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:59.647920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.193:36245)
0621 07:49:59.648419 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:59.648479 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0621 07:49:59.650025 (+ 1546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:49:59.650518 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:49:59.650524 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:49:59.651031 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:49:59.651689 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:59.651711 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:59.652613 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:59.652620 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:59.654226 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:59.654241 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:59.654756 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:59.654763 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:59.654988 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:59.655631 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:49:59.655654 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:49:59.657379 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:49:59.659252 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:59.659258 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:59.659274 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:59.659623 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:59.659924 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:59.659926 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:59.659928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:59.660051 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:59.660518 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:49:59.660523 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:49:59.660809 (+ 286us) client_negotiation.cc:769] Sending connection context
0621 07:49:59.660993 (+ 184us) client_negotiation.cc:240] Negotiation successful
0621 07:49:59.661307 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":103,"threads_started":1}
I20260621 07:49:59.663539 16358 heartbeater.cc:347] Connected to a master server at 127.0.192.254:45149
I20260621 07:49:59.663805 16358 heartbeater.cc:464] Registering TS with master...
I20260621 07:49:59.664299 16358 heartbeater.cc:511] Master 127.0.192.254:45149 requested a full tablet report, sending...
I20260621 07:49:59.666013 16167 ts_manager.cc:194] Registered new tserver with Master: 854c12bbec8949f397347e2dc7604940 (127.0.192.193:46373)
I20260621 07:49:59.667543 16167 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.193@KRBTEST.COM'} at 127.0.192.193:36245
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab.
Entry for principal kudu/127.0.192.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab.
I20260621 07:49:59.673810 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.194:0
--local_ip_for_outbound_sockets=127.0.192.194
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:45149
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:35349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:49:59.781929 16363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:49:59.782159 16363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:49:59.782199 16363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:49:59.786010 16363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:49:59.786062 16363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:49:59.786115 16363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.194
I20260621 07:49:59.791376 16363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:35349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:45149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.194
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:49:59.792397 16363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:49:59.793443 16363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 07:49:59.802868 16363 server_base.cc:1034] running on GCE node
W20260621 07:49:59.802835 16368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:49:59.802838 16371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:49:59.802835 16369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:49:59.803485 16363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:49:59.804039 16363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:49:59.805191 16363 hybrid_clock.cc:648] HybridClock initialized: now 1782028199805178 us; error 27 us; skew 500 ppm
Jun 21 07:49:59 dist-test-slave-lzx3 krb5kdc[16122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:49:59.807783 16363 init.cc:377] Logged in from keytab as kudu/127.0.192.194@KRBTEST.COM (short username kudu)
I20260621 07:49:59.808686 16363 webserver.cc:460] Webserver started at http://127.0.192.194:44757/ using document root <none> and password file <none>
I20260621 07:49:59.809222 16363 fs_manager.cc:359] Metadata directory not provided
I20260621 07:49:59.809310 16363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:49:59.809521 16363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:49:59.811523 16363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/data/instance:
uuid: "8737bbdc600e4cfbaa1fbc71a606db0f"
format_stamp: "Formatted at 2026-06-21 07:49:59 on dist-test-slave-lzx3"
server_key: "f9fd2a9d6bee71d733687e4d29c840b2"
server_key_iv: "e5a4a3aa53f03eaa9be9c286129e59fb"
server_key_version: "encryptionkey@0"
I20260621 07:49:59.812078 16363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/wal/instance:
uuid: "8737bbdc600e4cfbaa1fbc71a606db0f"
format_stamp: "Formatted at 2026-06-21 07:49:59 on dist-test-slave-lzx3"
server_key: "f9fd2a9d6bee71d733687e4d29c840b2"
server_key_iv: "e5a4a3aa53f03eaa9be9c286129e59fb"
server_key_version: "encryptionkey@0"
I20260621 07:49:59.815570 16363 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260621 07:49:59.817833 16378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:59.819000 16363 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 07:49:59.819137 16363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/wal
uuid: "8737bbdc600e4cfbaa1fbc71a606db0f"
format_stamp: "Formatted at 2026-06-21 07:49:59 on dist-test-slave-lzx3"
server_key: "f9fd2a9d6bee71d733687e4d29c840b2"
server_key_iv: "e5a4a3aa53f03eaa9be9c286129e59fb"
server_key_version: "encryptionkey@0"
I20260621 07:49:59.819247 16363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:49:59.839625 16363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:49:59.842836 16363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:49:59.843024 16363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:49:59.843568 16363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:49:59.844332 16363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:49:59.844405 16363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:59.844477 16363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:49:59.844527 16363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:49:59.853107 16363 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.194:43481
I20260621 07:49:59.853130 16491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.194:43481 every 8 connection(s)
I20260621 07:49:59.853974 16363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
I20260621 07:49:59.860690 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 16363
I20260621 07:49:59.860822 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/wal/instance
I20260621 07:49:59.861114 771 external_mini_cluster.cc:1428] Setting key d3d700b741c45bfd1942546703e26a98
Jun 21 07:49:59 dist-test-slave-lzx3 krb5kdc[16122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:49:59.868310 16206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:59.855628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:60177 (local address 127.0.192.254:45149)
0621 07:49:59.855797 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0621 07:49:59.855802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:49:59.856543 (+ 741us) server_negotiation.cc:407] Connection header received
0621 07:49:59.857805 (+ 1262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:49:59.857808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:49:59.857855 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:49:59.857931 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:49:59.859089 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:59.859597 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:59.860920 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:49:59.861143 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:49:59.861293 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:59.863900 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:49:59.863903 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:49:59.863905 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:49:59.863929 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:49:59.865677 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:59.866329 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:59.866332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:59.866334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:59.866480 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:49:59.866916 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:49:59.866919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:49:59.866921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:49:59.867199 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:49:59.867295 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0621 07:49:59.868009 (+ 714us) server_negotiation.cc:299] Negotiation successful
0621 07:49:59.868131 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 07:49:59.869222 16494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:49:59.856004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.194:60177)
0621 07:49:59.856399 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0621 07:49:59.856437 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 07:49:59.857593 (+ 1156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:49:59.858120 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:49:59.858127 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:49:59.858502 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:49:59.858960 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:59.858973 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:59.859716 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:59.859719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:59.860806 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:49:59.860818 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:49:59.861329 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:49:59.861333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:49:59.861502 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:49:59.862085 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:49:59.862100 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:49:59.863731 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:49:59.865810 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:59.865815 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:59.865830 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:59.866213 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:59.866595 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:49:59.866598 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:49:59.866600 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:49:59.866806 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:49:59.867318 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:49:59.867324 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:49:59.867688 (+ 364us) client_negotiation.cc:769] Sending connection context
0621 07:49:59.867935 (+ 247us) client_negotiation.cc:240] Negotiation successful
0621 07:49:59.868281 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":100,"threads_started":1}
I20260621 07:49:59.870545 16492 heartbeater.cc:347] Connected to a master server at 127.0.192.254:45149
I20260621 07:49:59.870822 16492 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.192.195@KRBTEST.COM; defaulting to no policy
I20260621 07:49:59.871341 16492 heartbeater.cc:511] Master 127.0.192.254:45149 requested a full tablet report, sending...
I20260621 07:49:59.872399 16167 ts_manager.cc:194] Registered new tserver with Master: 8737bbdc600e4cfbaa1fbc71a606db0f (127.0.192.194:43481)
I20260621 07:49:59.873056 16167 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.194@KRBTEST.COM'} at 127.0.192.194:60177
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab.
Entry for principal kudu/127.0.192.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab.
I20260621 07:49:59.887755 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.195:0
--local_ip_for_outbound_sockets=127.0.192.195
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:45149
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:35349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:49:59.999893 16497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:50:00.000166 16497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:50:00.000252 16497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:50:00.004216 16497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:50:00.004302 16497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:50:00.004386 16497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.195
I20260621 07:50:00.009673 16497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:35349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:45149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.195
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:50:00.010672 16497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:50:00.011664 16497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:50:00.020788 16503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:50:00.020784 16502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:50:00.021145 16497 server_base.cc:1034] running on GCE node
W20260621 07:50:00.020816 16505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:50:00.022488 16497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:50:00.023116 16497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:50:00.024296 16497 hybrid_clock.cc:648] HybridClock initialized: now 1782028200024276 us; error 43 us; skew 500 ppm
Jun 21 07:50:00 dist-test-slave-lzx3 krb5kdc[16122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:50:00.027297 16497 init.cc:377] Logged in from keytab as kudu/127.0.192.195@KRBTEST.COM (short username kudu)
I20260621 07:50:00.028332 16497 webserver.cc:460] Webserver started at http://127.0.192.195:32853/ using document root <none> and password file <none>
I20260621 07:50:00.028942 16497 fs_manager.cc:359] Metadata directory not provided
I20260621 07:50:00.029029 16497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:50:00.029229 16497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:50:00.031216 16497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/data/instance:
uuid: "48ffd15f6b16463ebd049ab624d11205"
format_stamp: "Formatted at 2026-06-21 07:50:00 on dist-test-slave-lzx3"
server_key: "d194414a0870e7e7d2f427192faade69"
server_key_iv: "b241a51556df352e2bcadbc68b5de72a"
server_key_version: "encryptionkey@0"
I20260621 07:50:00.031781 16497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/wal/instance:
uuid: "48ffd15f6b16463ebd049ab624d11205"
format_stamp: "Formatted at 2026-06-21 07:50:00 on dist-test-slave-lzx3"
server_key: "d194414a0870e7e7d2f427192faade69"
server_key_iv: "b241a51556df352e2bcadbc68b5de72a"
server_key_version: "encryptionkey@0"
I20260621 07:50:00.035481 16497 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260621 07:50:00.037899 16512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:50:00.039152 16497 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 07:50:00.039264 16497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/wal
uuid: "48ffd15f6b16463ebd049ab624d11205"
format_stamp: "Formatted at 2026-06-21 07:50:00 on dist-test-slave-lzx3"
server_key: "d194414a0870e7e7d2f427192faade69"
server_key_iv: "b241a51556df352e2bcadbc68b5de72a"
server_key_version: "encryptionkey@0"
I20260621 07:50:00.039367 16497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:50:00.057189 16497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:50:00.060204 16497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:50:00.060396 16497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:50:00.061056 16497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:50:00.061895 16497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:50:00.061946 16497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:50:00.062012 16497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:50:00.062049 16497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:50:00.072229 16497 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.195:45517
I20260621 07:50:00.072250 16625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.195:45517 every 8 connection(s)
I20260621 07:50:00.073154 16497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
I20260621 07:50:00.076376 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 16497
I20260621 07:50:00.076499 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/wal/instance
I20260621 07:50:00.076838 771 external_mini_cluster.cc:1428] Setting key fbbe6b60225acdcdf8de0d330580f443
Jun 21 07:50:00 dist-test-slave-lzx3 krb5kdc[16122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:50:00.086042 16206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:00.074960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:39929 (local address 127.0.192.254:45149)
0621 07:50:00.075116 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0621 07:50:00.075120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:50:00.076355 (+ 1235us) server_negotiation.cc:407] Connection header received
0621 07:50:00.077404 (+ 1049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:50:00.077407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:50:00.077466 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:50:00.077580 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:50:00.078672 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:00.079205 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:00.080297 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:00.080514 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:00.080668 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:00.082820 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:50:00.082823 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:50:00.082826 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:50:00.082856 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:50:00.084058 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:50:00.084535 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:50:00.084539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:50:00.084541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:50:00.084667 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:50:00.085013 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:50:00.085016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:50:00.085018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:50:00.085193 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:50:00.085301 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0621 07:50:00.085760 (+ 459us) server_negotiation.cc:299] Negotiation successful
0621 07:50:00.085876 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260621 07:50:00.086762 16628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:00.075345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.195:39929)
0621 07:50:00.076212 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:00.076251 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0621 07:50:00.077206 (+ 955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:50:00.077740 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:50:00.077746 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:50:00.078085 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:50:00.078534 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:00.078547 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:00.079327 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:00.079338 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:00.080177 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:00.080185 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:00.080700 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:00.080704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:00.080862 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:00.081359 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:50:00.081373 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:50:00.082667 (+ 1294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:50:00.084169 (+ 1502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:50:00.084174 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:50:00.084187 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:50:00.084434 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:50:00.084759 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:50:00.084763 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:50:00.084765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:50:00.084894 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:50:00.085301 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:50:00.085308 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:50:00.085536 (+ 228us) client_negotiation.cc:769] Sending connection context
0621 07:50:00.085721 (+ 185us) client_negotiation.cc:240] Negotiation successful
0621 07:50:00.086015 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"thread_start_us":552,"threads_started":1}
I20260621 07:50:00.087795 16626 heartbeater.cc:347] Connected to a master server at 127.0.192.254:45149
I20260621 07:50:00.087987 16626 heartbeater.cc:464] Registering TS with master...
I20260621 07:50:00.088392 16626 heartbeater.cc:511] Master 127.0.192.254:45149 requested a full tablet report, sending...
I20260621 07:50:00.089385 16167 ts_manager.cc:194] Registered new tserver with Master: 48ffd15f6b16463ebd049ab624d11205 (127.0.192.195:45517)
I20260621 07:50:00.090073 16167 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.195@KRBTEST.COM'} at 127.0.192.195:39929
I20260621 07:50:00.090521 771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:50:00.099843 16206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:00.092701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60366 (local address 127.0.192.254:45149)
0621 07:50:00.092842 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0621 07:50:00.092846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:50:00.093038 (+ 192us) server_negotiation.cc:407] Connection header received
0621 07:50:00.093237 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:50:00.093240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:50:00.093286 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:50:00.093368 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:50:00.094043 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:00.094535 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:00.095431 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:00.095643 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:00.095743 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:00.096964 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:50:00.096967 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:50:00.096970 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:50:00.096999 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:50:00.098214 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:50:00.098620 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:50:00.098625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:50:00.098627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:50:00.098704 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:50:00.098976 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:50:00.098979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:50:00.098981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:50:00.099174 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:50:00.099324 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0621 07:50:00.099567 (+ 243us) server_negotiation.cc:299] Negotiation successful
0621 07:50:00.099678 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260621 07:50:00.102375 16167 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60366:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:50:00.105480 16167 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:50:00.119489 16639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:00.113784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:43481 (local address 127.0.0.1:52606)
0621 07:50:00.114538 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:00.114553 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:50:00.114634 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:50:00.115398 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:50:00.115401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:50:00.115419 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:50:00.115546 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:00.115553 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:00.116878 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:00.116885 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:00.118054 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:00.118065 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:00.119120 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:00.119124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:00.119204 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:00.119223 (+ 19us) client_negotiation.cc:769] Sending connection context
0621 07:50:00.119290 (+ 67us) client_negotiation.cc:240] Negotiation successful
0621 07:50:00.119341 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"mutex_wait_us":19,"thread_start_us":144,"threads_started":1}
I20260621 07:50:00.119560 16641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:00.114376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52606 (local address 127.0.192.194:43481)
0621 07:50:00.114897 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0621 07:50:00.114910 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 07:50:00.114933 (+ 23us) server_negotiation.cc:407] Connection header received
0621 07:50:00.115005 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:50:00.115020 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:50:00.115218 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:50:00.115407 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:50:00.115664 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:00.116741 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:00.118190 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:00.119005 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:00.119106 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:00.119231 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0621 07:50:00.119313 (+ 82us) server_negotiation.cc:299] Negotiation successful
0621 07:50:00.119409 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":121,"threads_started":1}
I20260621 07:50:00.119390 16637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:00.113596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:46373 (local address 127.0.0.1:40452)
0621 07:50:00.113962 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:00.114012 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0621 07:50:00.114129 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:50:00.115069 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:50:00.115075 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:50:00.115098 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:50:00.115231 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:00.115239 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:00.116726 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:00.116730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:00.118015 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:00.118036 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:00.118823 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:00.118826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:00.118937 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:00.118974 (+ 37us) client_negotiation.cc:769] Sending connection context
0621 07:50:00.119109 (+ 135us) client_negotiation.cc:240] Negotiation successful
0621 07:50:00.119225 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":56,"thread_start_us":94,"threads_started":1}
I20260621 07:50:00.119732 16638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:00.113757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40452 (local address 127.0.192.193:46373)
0621 07:50:00.114332 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0621 07:50:00.114340 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:50:00.114365 (+ 25us) server_negotiation.cc:407] Connection header received
0621 07:50:00.114448 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:50:00.114454 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:50:00.114694 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:50:00.114858 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:50:00.115357 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:00.116602 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:00.118144 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:00.118703 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:00.118826 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:00.118978 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0621 07:50:00.119465 (+ 487us) server_negotiation.cc:299] Negotiation successful
0621 07:50:00.119571 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":72,"threads_started":1}
I20260621 07:50:00.120610 16640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:00.114372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.195:45517 (local address 127.0.0.1:42956)
0621 07:50:00.115410 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:00.115423 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:50:00.115499 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:50:00.115918 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:50:00.115921 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:50:00.115937 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:50:00.116017 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:00.116023 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:00.117474 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:00.117477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:00.118730 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:00.118739 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:00.120205 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:00.120208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:00.120291 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:00.120314 (+ 23us) client_negotiation.cc:769] Sending connection context
0621 07:50:00.120393 (+ 79us) client_negotiation.cc:240] Negotiation successful
0621 07:50:00.120456 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":980,"mutex_wait_us":48,"thread_start_us":84,"threads_started":1}
I20260621 07:50:00.121044 16642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:00.114751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42956 (local address 127.0.192.195:45517)
0621 07:50:00.115272 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0621 07:50:00.115286 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 07:50:00.115590 (+ 304us) server_negotiation.cc:407] Connection header received
0621 07:50:00.115673 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:50:00.115677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:50:00.115798 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:50:00.115938 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:50:00.116106 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:00.116943 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:00.118879 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:00.120084 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:00.120190 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:00.120313 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0621 07:50:00.120429 (+ 116us) server_negotiation.cc:299] Negotiation successful
0621 07:50:00.120513 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":159,"threads_started":1}
I20260621 07:50:00.122728 16292 tablet_service.cc:1467] Processing CreateTablet for tablet 7950b31cbf95472780fa872439374c9a (DEFAULT_TABLE table=test-table [id=fd374edb15834fe39fb5ea8875535ecd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:50:00.122786 16426 tablet_service.cc:1467] Processing CreateTablet for tablet 7950b31cbf95472780fa872439374c9a (DEFAULT_TABLE table=test-table [id=fd374edb15834fe39fb5ea8875535ecd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:50:00.123760 16292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7950b31cbf95472780fa872439374c9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:50:00.123795 16426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7950b31cbf95472780fa872439374c9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:50:00.123715 16560 tablet_service.cc:1467] Processing CreateTablet for tablet 7950b31cbf95472780fa872439374c9a (DEFAULT_TABLE table=test-table [id=fd374edb15834fe39fb5ea8875535ecd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:50:00.124665 16560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7950b31cbf95472780fa872439374c9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:50:00.130260 16643 tablet_bootstrap.cc:492] T 7950b31cbf95472780fa872439374c9a P 854c12bbec8949f397347e2dc7604940: Bootstrap starting.
I20260621 07:50:00.131771 16645 tablet_bootstrap.cc:492] T 7950b31cbf95472780fa872439374c9a P 48ffd15f6b16463ebd049ab624d11205: Bootstrap starting.
I20260621 07:50:00.132762 16643 tablet_bootstrap.cc:654] T 7950b31cbf95472780fa872439374c9a P 854c12bbec8949f397347e2dc7604940: Neither blocks nor log segments found. Creating new log.
I20260621 07:50:00.132887 16644 tablet_bootstrap.cc:492] T 7950b31cbf95472780fa872439374c9a P 8737bbdc600e4cfbaa1fbc71a606db0f: Bootstrap starting.
I20260621 07:50:00.133531 16643 log.cc:824] T 7950b31cbf95472780fa872439374c9a P 854c12bbec8949f397347e2dc7604940: Log is configured to *not* fsync() on all Append() calls
I20260621 07:50:00.133762 16645 tablet_bootstrap.cc:654] T 7950b31cbf95472780fa872439374c9a P 48ffd15f6b16463ebd049ab624d11205: Neither blocks nor log segments found. Creating new log.
I20260621 07:50:00.134925 16645 log.cc:824] T 7950b31cbf95472780fa872439374c9a P 48ffd15f6b16463ebd049ab624d11205: Log is configured to *not* fsync() on all Append() calls
I20260621 07:50:00.135385 16644 tablet_bootstrap.cc:654] T 7950b31cbf95472780fa872439374c9a P 8737bbdc600e4cfbaa1fbc71a606db0f: Neither blocks nor log segments found. Creating new log.
I20260621 07:50:00.136184 16644 log.cc:824] T 7950b31cbf95472780fa872439374c9a P 8737bbdc600e4cfbaa1fbc71a606db0f: Log is configured to *not* fsync() on all Append() calls
I20260621 07:50:00.137180 16645 tablet_bootstrap.cc:492] T 7950b31cbf95472780fa872439374c9a P 48ffd15f6b16463ebd049ab624d11205: No bootstrap required, opened a new log
I20260621 07:50:00.137390 16645 ts_tablet_manager.cc:1397] T 7950b31cbf95472780fa872439374c9a P 48ffd15f6b16463ebd049ab624d11205: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260621 07:50:00.138049 16644 tablet_bootstrap.cc:492] T 7950b31cbf95472780fa872439374c9a P 8737bbdc600e4cfbaa1fbc71a606db0f: No bootstrap required, opened a new log
I20260621 07:50:00.138060 16643 tablet_bootstrap.cc:492] T 7950b31cbf95472780fa872439374c9a P 854c12bbec8949f397347e2dc7604940: No bootstrap required, opened a new log
I20260621 07:50:00.138238 16644 ts_tablet_manager.cc:1397] T 7950b31cbf95472780fa872439374c9a P 8737bbdc600e4cfbaa1fbc71a606db0f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260621 07:50:00.138238 16643 ts_tablet_manager.cc:1397] T 7950b31cbf95472780fa872439374c9a P 854c12bbec8949f397347e2dc7604940: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260621 07:50:00.140260 16645 raft_consensus.cc:348] T 7950b31cbf95472780fa872439374c9a P 48ffd15f6b16463ebd049ab624d11205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854c12bbec8949f397347e2dc7604940" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46373 } } peers { permanent_uuid: "8737bbdc600e4cfbaa1fbc71a606db0f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43481 } } peers { permanent_uuid: "48ffd15f6b16463ebd049ab624d11205" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 45517 } }
I20260621 07:50:00.140516 16645 raft_consensus.cc:374] T 7950b31cbf95472780fa872439374c9a P 48ffd15f6b16463ebd049ab624d11205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:50:00.140611 16645 raft_consensus.cc:729] T 7950b31cbf95472780fa872439374c9a P 48ffd15f6b16463ebd049ab624d11205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48ffd15f6b16463ebd049ab624d11205, State: Initialized, Role: FOLLOWER
I20260621 07:50:00.141080 16645 consensus_queue.cc:260] T 7950b31cbf95472780fa872439374c9a P 48ffd15f6b16463ebd049ab624d11205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854c12bbec8949f397347e2dc7604940" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46373 } } peers { permanent_uuid: "8737bbdc600e4cfbaa1fbc71a606db0f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43481 } } peers { permanent_uuid: "48ffd15f6b16463ebd049ab624d11205" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 45517 } }
I20260621 07:50:00.141697 16626 heartbeater.cc:503] Master 127.0.192.254:45149 was elected leader, sending a full tablet report...
I20260621 07:50:00.142052 16645 ts_tablet_manager.cc:1428] T 7950b31cbf95472780fa872439374c9a P 48ffd15f6b16463ebd049ab624d11205: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260621 07:50:00.142031 16644 raft_consensus.cc:348] T 7950b31cbf95472780fa872439374c9a P 8737bbdc600e4cfbaa1fbc71a606db0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854c12bbec8949f397347e2dc7604940" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46373 } } peers { permanent_uuid: "8737bbdc600e4cfbaa1fbc71a606db0f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43481 } } peers { permanent_uuid: "48ffd15f6b16463ebd049ab624d11205" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 45517 } }
I20260621 07:50:00.142297 16644 raft_consensus.cc:374] T 7950b31cbf95472780fa872439374c9a P 8737bbdc600e4cfbaa1fbc71a606db0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:50:00.142364 16644 raft_consensus.cc:729] T 7950b31cbf95472780fa872439374c9a P 8737bbdc600e4cfbaa1fbc71a606db0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8737bbdc600e4cfbaa1fbc71a606db0f, State: Initialized, Role: FOLLOWER
I20260621 07:50:00.142781 16644 consensus_queue.cc:260] T 7950b31cbf95472780fa872439374c9a P 8737bbdc600e4cfbaa1fbc71a606db0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854c12bbec8949f397347e2dc7604940" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46373 } } peers { permanent_uuid: "8737bbdc600e4cfbaa1fbc71a606db0f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43481 } } peers { permanent_uuid: "48ffd15f6b16463ebd049ab624d11205" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 45517 } }
I20260621 07:50:00.142976 16643 raft_consensus.cc:348] T 7950b31cbf95472780fa872439374c9a P 854c12bbec8949f397347e2dc7604940 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854c12bbec8949f397347e2dc7604940" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46373 } } peers { permanent_uuid: "8737bbdc600e4cfbaa1fbc71a606db0f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43481 } } peers { permanent_uuid: "48ffd15f6b16463ebd049ab624d11205" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 45517 } }
I20260621 07:50:00.143280 16643 raft_consensus.cc:374] T 7950b31cbf95472780fa872439374c9a P 854c12bbec8949f397347e2dc7604940 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:50:00.143366 16643 raft_consensus.cc:729] T 7950b31cbf95472780fa872439374c9a P 854c12bbec8949f397347e2dc7604940 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 854c12bbec8949f397347e2dc7604940, State: Initialized, Role: FOLLOWER
I20260621 07:50:00.143539 16492 heartbeater.cc:503] Master 127.0.192.254:45149 was elected leader, sending a full tablet report...
I20260621 07:50:00.143776 16644 ts_tablet_manager.cc:1428] T 7950b31cbf95472780fa872439374c9a P 8737bbdc600e4cfbaa1fbc71a606db0f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260621 07:50:00.143806 16643 consensus_queue.cc:260] T 7950b31cbf95472780fa872439374c9a P 854c12bbec8949f397347e2dc7604940 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854c12bbec8949f397347e2dc7604940" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46373 } } peers { permanent_uuid: "8737bbdc600e4cfbaa1fbc71a606db0f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43481 } } peers { permanent_uuid: "48ffd15f6b16463ebd049ab624d11205" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 45517 } }
I20260621 07:50:00.144742 16643 ts_tablet_manager.cc:1428] T 7950b31cbf95472780fa872439374c9a P 854c12bbec8949f397347e2dc7604940: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260621 07:50:00.145048 16358 heartbeater.cc:503] Master 127.0.192.254:45149 was elected leader, sending a full tablet report...
W20260621 07:50:00.153725 16359 tablet.cc:2369] T 7950b31cbf95472780fa872439374c9a P 854c12bbec8949f397347e2dc7604940: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:50:00.174063 16650 raft_consensus.cc:482] T 7950b31cbf95472780fa872439374c9a P 8737bbdc600e4cfbaa1fbc71a606db0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:50:00.174258 16650 raft_consensus.cc:504] T 7950b31cbf95472780fa872439374c9a P 8737bbdc600e4cfbaa1fbc71a606db0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854c12bbec8949f397347e2dc7604940" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46373 } } peers { permanent_uuid: "8737bbdc600e4cfbaa1fbc71a606db0f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43481 } } peers { permanent_uuid: "48ffd15f6b16463ebd049ab624d11205" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 45517 } }
I20260621 07:50:00.175420 16650 leader_election.cc:288] T 7950b31cbf95472780fa872439374c9a P 8737bbdc600e4cfbaa1fbc71a606db0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 854c12bbec8949f397347e2dc7604940 (127.0.192.193:46373), 48ffd15f6b16463ebd049ab624d11205 (127.0.192.195:45517)
I20260621 07:50:00.178975 16494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:00.175638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:46373 (local address 127.0.192.194:46689)
0621 07:50:00.175833 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:00.175851 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:50:00.175955 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:50:00.176247 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:50:00.176250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:50:00.176266 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:50:00.176401 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:00.176407 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:00.177170 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:00.177173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:00.177912 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:00.177920 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:00.178564 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:00.178567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:00.178669 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:00.178692 (+ 23us) client_negotiation.cc:769] Sending connection context
0621 07:50:00.178784 (+ 92us) client_negotiation.cc:240] Negotiation successful
0621 07:50:00.178852 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"spinlock_wait_cycles":34560}
I20260621 07:50:00.178975 16638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:00.175712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:46689 (local address 127.0.192.193:46373)
0621 07:50:00.175833 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0621 07:50:00.175840 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:50:00.175909 (+ 69us) server_negotiation.cc:407] Connection header received
0621 07:50:00.176080 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:50:00.176083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:50:00.176135 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:50:00.176234 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:50:00.176519 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:00.177058 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:00.178038 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:00.178436 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:00.178565 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:00.178617 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0621 07:50:00.178798 (+ 181us) server_negotiation.cc:299] Negotiation successful
0621 07:50:00.178861 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260621 07:50:00.179471 16642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:00.175780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:34349 (local address 127.0.192.195:45517)
0621 07:50:00.175922 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0621 07:50:00.175927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:50:00.176297 (+ 370us) server_negotiation.cc:407] Connection header received
0621 07:50:00.176452 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:50:00.176456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:50:00.176502 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:50:00.176643 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:50:00.176926 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:00.177418 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:00.178471 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:00.178894 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:00.179032 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:00.179084 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0621 07:50:00.179280 (+ 196us) server_negotiation.cc:299] Negotiation successful
0621 07:50:00.179349 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 07:50:00.179498 16652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:00.175643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.195:45517 (local address 127.0.192.194:34349)
0621 07:50:00.176191 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:00.176201 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 07:50:00.176332 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:50:00.176658 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:50:00.176661 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:50:00.176677 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:50:00.176813 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:00.176818 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:00.177540 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:00.177544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:00.178331 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:00.178340 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:00.179025 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:00.179029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:00.179131 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:00.179154 (+ 23us) client_negotiation.cc:769] Sending connection context
0621 07:50:00.179259 (+ 105us) client_negotiation.cc:240] Negotiation successful
0621 07:50:00.179330 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":213,"spinlock_wait_cycles":768,"thread_start_us":152,"threads_started":1}
I20260621 07:50:00.179837 16312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7950b31cbf95472780fa872439374c9a" candidate_uuid: "8737bbdc600e4cfbaa1fbc71a606db0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "854c12bbec8949f397347e2dc7604940" is_pre_election: true
I20260621 07:50:00.180119 16312 raft_consensus.cc:2454] T 7950b31cbf95472780fa872439374c9a P 854c12bbec8949f397347e2dc7604940 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8737bbdc600e4cfbaa1fbc71a606db0f in term 0.
I20260621 07:50:00.180039 16580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7950b31cbf95472780fa872439374c9a" candidate_uuid: "8737bbdc600e4cfbaa1fbc71a606db0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48ffd15f6b16463ebd049ab624d11205" is_pre_election: true
I20260621 07:50:00.180370 16580 raft_consensus.cc:2454] T 7950b31cbf95472780fa872439374c9a P 48ffd15f6b16463ebd049ab624d11205 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8737bbdc600e4cfbaa1fbc71a606db0f in term 0.
I20260621 07:50:00.180637 16379 leader_election.cc:302] T 7950b31cbf95472780fa872439374c9a P 8737bbdc600e4cfbaa1fbc71a606db0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 854c12bbec8949f397347e2dc7604940, 8737bbdc600e4cfbaa1fbc71a606db0f; no voters:
I20260621 07:50:00.180935 16650 raft_consensus.cc:2781] T 7950b31cbf95472780fa872439374c9a P 8737bbdc600e4cfbaa1fbc71a606db0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:50:00.181007 16650 raft_consensus.cc:482] T 7950b31cbf95472780fa872439374c9a P 8737bbdc600e4cfbaa1fbc71a606db0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:50:00.181036 16650 raft_consensus.cc:3037] T 7950b31cbf95472780fa872439374c9a P 8737bbdc600e4cfbaa1fbc71a606db0f [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:50:00.182031 16650 raft_consensus.cc:504] T 7950b31cbf95472780fa872439374c9a P 8737bbdc600e4cfbaa1fbc71a606db0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854c12bbec8949f397347e2dc7604940" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46373 } } peers { permanent_uuid: "8737bbdc600e4cfbaa1fbc71a606db0f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43481 } } peers { permanent_uuid: "48ffd15f6b16463ebd049ab624d11205" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 45517 } }
I20260621 07:50:00.182374 16650 leader_election.cc:288] T 7950b31cbf95472780fa872439374c9a P 8737bbdc600e4cfbaa1fbc71a606db0f [CANDIDATE]: Term 1 election: Requested vote from peers 854c12bbec8949f397347e2dc7604940 (127.0.192.193:46373), 48ffd15f6b16463ebd049ab624d11205 (127.0.192.195:45517)
I20260621 07:50:00.182744 16580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7950b31cbf95472780fa872439374c9a" candidate_uuid: "8737bbdc600e4cfbaa1fbc71a606db0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48ffd15f6b16463ebd049ab624d11205"
I20260621 07:50:00.182765 16312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7950b31cbf95472780fa872439374c9a" candidate_uuid: "8737bbdc600e4cfbaa1fbc71a606db0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "854c12bbec8949f397347e2dc7604940"
I20260621 07:50:00.182860 16580 raft_consensus.cc:3037] T 7950b31cbf95472780fa872439374c9a P 48ffd15f6b16463ebd049ab624d11205 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:50:00.182860 16312 raft_consensus.cc:3037] T 7950b31cbf95472780fa872439374c9a P 854c12bbec8949f397347e2dc7604940 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:50:00.184052 16580 raft_consensus.cc:2454] T 7950b31cbf95472780fa872439374c9a P 48ffd15f6b16463ebd049ab624d11205 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8737bbdc600e4cfbaa1fbc71a606db0f in term 1.
I20260621 07:50:00.184052 16312 raft_consensus.cc:2454] T 7950b31cbf95472780fa872439374c9a P 854c12bbec8949f397347e2dc7604940 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8737bbdc600e4cfbaa1fbc71a606db0f in term 1.
I20260621 07:50:00.184396 16380 leader_election.cc:302] T 7950b31cbf95472780fa872439374c9a P 8737bbdc600e4cfbaa1fbc71a606db0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48ffd15f6b16463ebd049ab624d11205, 8737bbdc600e4cfbaa1fbc71a606db0f; no voters:
I20260621 07:50:00.184633 16650 raft_consensus.cc:2781] T 7950b31cbf95472780fa872439374c9a P 8737bbdc600e4cfbaa1fbc71a606db0f [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:50:00.184844 16650 raft_consensus.cc:686] T 7950b31cbf95472780fa872439374c9a P 8737bbdc600e4cfbaa1fbc71a606db0f [term 1 LEADER]: Becoming Leader. State: Replica: 8737bbdc600e4cfbaa1fbc71a606db0f, State: Running, Role: LEADER
I20260621 07:50:00.185153 16650 consensus_queue.cc:237] T 7950b31cbf95472780fa872439374c9a P 8737bbdc600e4cfbaa1fbc71a606db0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854c12bbec8949f397347e2dc7604940" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46373 } } peers { permanent_uuid: "8737bbdc600e4cfbaa1fbc71a606db0f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43481 } } peers { permanent_uuid: "48ffd15f6b16463ebd049ab624d11205" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 45517 } }
I20260621 07:50:00.188082 16166 catalog_manager.cc:5495] T 7950b31cbf95472780fa872439374c9a P 8737bbdc600e4cfbaa1fbc71a606db0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8737bbdc600e4cfbaa1fbc71a606db0f (127.0.192.194). New cstate: current_term: 1 leader_uuid: "8737bbdc600e4cfbaa1fbc71a606db0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854c12bbec8949f397347e2dc7604940" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8737bbdc600e4cfbaa1fbc71a606db0f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48ffd15f6b16463ebd049ab624d11205" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 45517 } health_report { overall_health: UNKNOWN } } }
I20260621 07:50:00.216945 16641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:00.213214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52618 (local address 127.0.192.194:43481)
0621 07:50:00.213374 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0621 07:50:00.213380 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:50:00.213395 (+ 15us) server_negotiation.cc:407] Connection header received
0621 07:50:00.213583 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:50:00.213585 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:50:00.213643 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:50:00.213768 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:50:00.214044 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:00.214560 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:00.215423 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:00.215635 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:00.215750 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:00.216073 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:50:00.216481 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:50:00.216646 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0621 07:50:00.216724 (+ 78us) server_negotiation.cc:299] Negotiation successful
0621 07:50:00.216790 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 07:50:00.222607 16580 raft_consensus.cc:1261] T 7950b31cbf95472780fa872439374c9a P 48ffd15f6b16463ebd049ab624d11205 [term 1 FOLLOWER]: Refusing update from remote peer 8737bbdc600e4cfbaa1fbc71a606db0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:50:00.222674 16312 raft_consensus.cc:1261] T 7950b31cbf95472780fa872439374c9a P 854c12bbec8949f397347e2dc7604940 [term 1 FOLLOWER]: Refusing update from remote peer 8737bbdc600e4cfbaa1fbc71a606db0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:50:00.223359 16653 consensus_queue.cc:1035] T 7950b31cbf95472780fa872439374c9a P 8737bbdc600e4cfbaa1fbc71a606db0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "48ffd15f6b16463ebd049ab624d11205" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 45517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:50:00.223563 16650 consensus_queue.cc:1035] T 7950b31cbf95472780fa872439374c9a P 8737bbdc600e4cfbaa1fbc71a606db0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "854c12bbec8949f397347e2dc7604940" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:50:00.230459 16656 mvcc.cc:205] Tried to move back new op lower bound from 7299187508105601024 to 7299187507961782272. Current Snapshot: MvccSnapshot[applied={T|T < 7299187508105601024}]
I20260621 07:50:00.232409 16657 mvcc.cc:205] Tried to move back new op lower bound from 7299187508105601024 to 7299187507961782272. Current Snapshot: MvccSnapshot[applied={T|T < 7299187508105601024 or (T in {7299187508105601024})}]
I20260621 07:50:00.239974 16166 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:50:00.240118 16166 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:50:00.242640 16166 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c: Sending DeleteTablet for 3 replicas of tablet 7950b31cbf95472780fa872439374c9a
I20260621 07:50:00.243412 16426 tablet_service.cc:1514] Processing DeleteTablet for tablet 7950b31cbf95472780fa872439374c9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:50:00 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:52606
I20260621 07:50:00.243407 16292 tablet_service.cc:1514] Processing DeleteTablet for tablet 7950b31cbf95472780fa872439374c9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:50:00 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:40452
I20260621 07:50:00.244154 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 16229
I20260621 07:50:00.244220 16560 tablet_service.cc:1514] Processing DeleteTablet for tablet 7950b31cbf95472780fa872439374c9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:50:00 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:42956
I20260621 07:50:00.243955 16667 tablet_replica.cc:330] T 7950b31cbf95472780fa872439374c9a P 8737bbdc600e4cfbaa1fbc71a606db0f: stopping tablet replica
I20260621 07:50:00.246069 16667 raft_consensus.cc:2229] T 7950b31cbf95472780fa872439374c9a P 8737bbdc600e4cfbaa1fbc71a606db0f [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:50:00.246471 16667 raft_consensus.cc:2258] T 7950b31cbf95472780fa872439374c9a P 8737bbdc600e4cfbaa1fbc71a606db0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:50:00.247776 16667 ts_tablet_manager.cc:1905] T 7950b31cbf95472780fa872439374c9a P 8737bbdc600e4cfbaa1fbc71a606db0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:50:00.250025 16667 ts_tablet_manager.cc:1918] T 7950b31cbf95472780fa872439374c9a P 8737bbdc600e4cfbaa1fbc71a606db0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:50:00.250145 16667 log.cc:1196] T 7950b31cbf95472780fa872439374c9a P 8737bbdc600e4cfbaa1fbc71a606db0f: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/wal/wals/7950b31cbf95472780fa872439374c9a
I20260621 07:50:00.250414 16668 tablet_replica.cc:330] T 7950b31cbf95472780fa872439374c9a P 48ffd15f6b16463ebd049ab624d11205: stopping tablet replica
I20260621 07:50:00.250552 16667 ts_tablet_manager.cc:1939] T 7950b31cbf95472780fa872439374c9a P 8737bbdc600e4cfbaa1fbc71a606db0f: Deleting consensus metadata
I20260621 07:50:00.250874 16668 raft_consensus.cc:2229] T 7950b31cbf95472780fa872439374c9a P 48ffd15f6b16463ebd049ab624d11205 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:50:00.251227 16668 raft_consensus.cc:2258] T 7950b31cbf95472780fa872439374c9a P 48ffd15f6b16463ebd049ab624d11205 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:50:00.252044 16153 catalog_manager.cc:4841] TS 8737bbdc600e4cfbaa1fbc71a606db0f (127.0.192.194:43481): tablet 7950b31cbf95472780fa872439374c9a (table test-table [id=fd374edb15834fe39fb5ea8875535ecd]) successfully deleted
W20260621 07:50:00.252957 16151 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.192.193:46373 (error 108)
I20260621 07:50:00.253181 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 16363
I20260621 07:50:00.253201 16668 ts_tablet_manager.cc:1905] T 7950b31cbf95472780fa872439374c9a P 48ffd15f6b16463ebd049ab624d11205: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:50:00.256515 16668 ts_tablet_manager.cc:1918] T 7950b31cbf95472780fa872439374c9a P 48ffd15f6b16463ebd049ab624d11205: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:50:00.256662 16668 log.cc:1196] T 7950b31cbf95472780fa872439374c9a P 48ffd15f6b16463ebd049ab624d11205: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/wal/wals/7950b31cbf95472780fa872439374c9a
I20260621 07:50:00.256979 16668 ts_tablet_manager.cc:1939] T 7950b31cbf95472780fa872439374c9a P 48ffd15f6b16463ebd049ab624d11205: Deleting consensus metadata
I20260621 07:50:00.257521 16637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:00.257149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:46373 (local address 127.0.0.1:40464)
0621 07:50:00.257310 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:00.257414 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.193:46373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260621 07:50:00.257748 16151 catalog_manager.cc:4584] TS 854c12bbec8949f397347e2dc7604940 (127.0.192.193:46373): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7950b31cbf95472780fa872439374c9a: Network error: Client connection negotiation failed: client connection to 127.0.192.193:46373: connect: Connection refused (error 111)
I20260621 07:50:00.258075 16152 catalog_manager.cc:4841] TS 48ffd15f6b16463ebd049ab624d11205 (127.0.192.195:45517): tablet 7950b31cbf95472780fa872439374c9a (table test-table [id=fd374edb15834fe39fb5ea8875535ecd]) successfully deleted
I20260621 07:50:00.259574 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 16497
I20260621 07:50:00.265419 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 16135
I20260621 07:50:20.272003 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:45149
--webserver_interface=127.0.192.254
--webserver_port=40327
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:35349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:45149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:50:20.366261 16670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:50:20.366456 16670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:50:20.366549 16670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:50:20.370055 16670 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260621 07:50:20.370090 16670 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260621 07:50:20.370105 16670 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260621 07:50:20.370117 16670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:50:20.370132 16670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:50:20.370148 16670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:50:20.370167 16670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:50:20.370180 16670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:50:20.374819 16670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:35349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:45149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:45149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=40327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:50:20.375553 16670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:50:20.376281 16670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:50:20.387444 16676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:50:20.387490 16670 server_base.cc:1034] running on GCE node
W20260621 07:50:20.387430 16679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:50:20.387446 16677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:50:20.388131 16670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:50:20.389130 16670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:50:20.390278 16670 hybrid_clock.cc:648] HybridClock initialized: now 1782028220390257 us; error 33 us; skew 500 ppm
Jun 21 07:50:20 dist-test-slave-lzx3 krb5kdc[16122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:50:20.392864 16670 init.cc:377] Logged in from keytab as kudu/127.0.192.254@KRBTEST.COM (short username kudu)
I20260621 07:50:20.393621 16670 webserver.cc:460] Webserver started at http://127.0.192.254:40327/ using document root <none> and password file <none>
I20260621 07:50:20.394104 16670 fs_manager.cc:359] Metadata directory not provided
I20260621 07:50:20.394186 16670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:50:20.397230 16670 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260621 07:50:20.398922 16686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:50:20.399881 16670 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260621 07:50:20.400002 16670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "98c78c73eb2247848e49787843c20f5c"
format_stamp: "Formatted at 2026-06-21 07:49:59 on dist-test-slave-lzx3"
server_key: "7f1307aaa489feea232904a5e07855ce"
server_key_iv: "a66aad73380eec7c6c8f0531051c12c0"
server_key_version: "encryptionkey@0"
I20260621 07:50:20.400457 16670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:50:20.412894 16670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:50:20.415033 16670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:50:20.415207 16670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:50:20.421074 16670 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:45149
I20260621 07:50:20.421339 16738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:45149 every 8 connection(s)
I20260621 07:50:20.422015 16670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:50:20.425745 16739 sys_catalog.cc:263] Verifying existing consensus state
I20260621 07:50:20.427136 16739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c: Bootstrap starting.
I20260621 07:50:20.427423 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 16670
I20260621 07:50:20.427875 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.193:46373
--local_ip_for_outbound_sockets=127.0.192.193
--tserver_master_addrs=127.0.192.254:45149
--webserver_port=42439
--webserver_interface=127.0.192.193
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:35349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:50:20.443012 16739 log.cc:824] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c: Log is configured to *not* fsync() on all Append() calls
I20260621 07:50:20.450435 16739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 07:50:20.450786 16739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c: Bootstrap complete.
I20260621 07:50:20.454414 16739 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c78c73eb2247848e49787843c20f5c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45149 } }
I20260621 07:50:20.454741 16739 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98c78c73eb2247848e49787843c20f5c, State: Initialized, Role: FOLLOWER
I20260621 07:50:20.455199 16739 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c78c73eb2247848e49787843c20f5c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45149 } }
I20260621 07:50:20.455368 16739 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:50:20.455482 16739 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:50:20.455595 16739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [term 1 FOLLOWER]: Advancing to term 2
I20260621 07:50:20.457317 16739 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c78c73eb2247848e49787843c20f5c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45149 } }
I20260621 07:50:20.457607 16739 leader_election.cc:302] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98c78c73eb2247848e49787843c20f5c; no voters:
I20260621 07:50:20.457875 16739 leader_election.cc:288] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260621 07:50:20.457931 16743 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [term 2 FOLLOWER]: Leader election won for term 2
I20260621 07:50:20.458117 16743 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [term 2 LEADER]: Becoming Leader. State: Replica: 98c78c73eb2247848e49787843c20f5c, State: Running, Role: LEADER
I20260621 07:50:20.458397 16743 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c78c73eb2247848e49787843c20f5c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45149 } }
I20260621 07:50:20.458914 16739 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:50:20.459193 16744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98c78c73eb2247848e49787843c20f5c. Latest consensus state: current_term: 2 leader_uuid: "98c78c73eb2247848e49787843c20f5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c78c73eb2247848e49787843c20f5c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45149 } } }
I20260621 07:50:20.459301 16744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [sys.catalog]: This master's current role is: LEADER
I20260621 07:50:20.459564 16745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "98c78c73eb2247848e49787843c20f5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c78c73eb2247848e49787843c20f5c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45149 } } }
I20260621 07:50:20.459681 16745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [sys.catalog]: This master's current role is: LEADER
I20260621 07:50:20.460292 16747 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:50:20.464541 16747 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:50:20.465165 16747 catalog_manager.cc:1259] Loaded cluster ID: 6831487686084c91b5de07c168ebca26
I20260621 07:50:20.465231 16747 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:50:20.467367 16747 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:50:20.467999 16747 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c: Loaded TSK: 0
I20260621 07:50:20.480762 16747 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c: Generated new TSK 1
I20260621 07:50:20.480973 16747 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 07:50:20.527878 16741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:50:20.528081 16741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:50:20.528151 16741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:50:20.531615 16741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:50:20.531690 16741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:50:20.531795 16741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.193
I20260621 07:50:20.536005 16741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:35349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.193:46373
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.192.193
--webserver_port=42439
--tserver_master_addrs=127.0.192.254:45149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.193
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:50:20.536777 16741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:50:20.537531 16741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:50:20.546490 16769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:50:20.546463 16766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:50:20.546619 16767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:50:20.546958 16741 server_base.cc:1034] running on GCE node
I20260621 07:50:20.547333 16741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:50:20.547968 16741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:50:20.549153 16741 hybrid_clock.cc:648] HybridClock initialized: now 1782028220549123 us; error 37 us; skew 500 ppm
Jun 21 07:50:20 dist-test-slave-lzx3 krb5kdc[16122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:50:20.552085 16741 init.cc:377] Logged in from keytab as kudu/127.0.192.193@KRBTEST.COM (short username kudu)
I20260621 07:50:20.552878 16741 webserver.cc:460] Webserver started at http://127.0.192.193:42439/ using document root <none> and password file <none>
I20260621 07:50:20.553367 16741 fs_manager.cc:359] Metadata directory not provided
I20260621 07:50:20.553476 16741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:50:20.557032 16741 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260621 07:50:20.558945 16776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:50:20.560087 16741 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:50:20.560191 16741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/wal
uuid: "854c12bbec8949f397347e2dc7604940"
format_stamp: "Formatted at 2026-06-21 07:49:59 on dist-test-slave-lzx3"
server_key: "c7258f611355283400ad1fb45aee31a1"
server_key_iv: "0585450526b4f4e32e317a48b8232c9c"
server_key_version: "encryptionkey@0"
I20260621 07:50:20.560789 16741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:50:20.572330 16741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:50:20.574998 16741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:50:20.575166 16741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:50:20.575655 16741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:50:20.576645 16783 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260621 07:50:20.580085 16741 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260621 07:50:20.580156 16741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260621 07:50:20.580196 16741 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260621 07:50:20.581754 16741 ts_tablet_manager.cc:610] Registered 1 tablets
I20260621 07:50:20.581808 16741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260621 07:50:20.582064 16783 tablet_bootstrap.cc:492] T 7950b31cbf95472780fa872439374c9a P 854c12bbec8949f397347e2dc7604940: Bootstrap starting.
I20260621 07:50:20.591976 16741 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.193:46373
I20260621 07:50:20.592003 16890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.193:46373 every 8 connection(s)
I20260621 07:50:20.593140 16741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
I20260621 07:50:20.593340 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 16741
I20260621 07:50:20.593775 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.194:43481
--local_ip_for_outbound_sockets=127.0.192.194
--tserver_master_addrs=127.0.192.254:45149
--webserver_port=44757
--webserver_interface=127.0.192.194
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:35349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:50:20.602043 16783 log.cc:824] T 7950b31cbf95472780fa872439374c9a P 854c12bbec8949f397347e2dc7604940: Log is configured to *not* fsync() on all Append() calls
Jun 21 07:50:20 dist-test-slave-lzx3 krb5kdc[16122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:50:20.609539 16783 tablet_bootstrap.cc:492] T 7950b31cbf95472780fa872439374c9a P 854c12bbec8949f397347e2dc7604940: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 07:50:20.609880 16783 tablet_bootstrap.cc:492] T 7950b31cbf95472780fa872439374c9a P 854c12bbec8949f397347e2dc7604940: Bootstrap complete.
I20260621 07:50:20.610884 16783 ts_tablet_manager.cc:1397] T 7950b31cbf95472780fa872439374c9a P 854c12bbec8949f397347e2dc7604940: Time spent bootstrapping tablet: real 0.029s user 0.025s sys 0.000s
I20260621 07:50:20.612977 16895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:20.596882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.193:41431)
0621 07:50:20.597416 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:20.597473 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0621 07:50:20.598459 (+ 986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:50:20.599799 (+ 1340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:50:20.599812 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:50:20.600302 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:50:20.600963 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:20.600981 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:20.602915 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:20.602919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:20.604160 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:20.604169 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:20.605290 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:20.605297 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:20.605632 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:20.606242 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:50:20.606259 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:50:20.607795 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:50:20.609909 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:50:20.609914 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:50:20.609925 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:50:20.610197 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:50:20.610561 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:50:20.610569 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:50:20.610571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:50:20.610730 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:50:20.611346 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:50:20.611352 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:50:20.611613 (+ 261us) client_negotiation.cc:769] Sending connection context
0621 07:50:20.611766 (+ 153us) client_negotiation.cc:240] Negotiation successful
0621 07:50:20.612039 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":134,"threads_started":1}
I20260621 07:50:20.612921 16783 raft_consensus.cc:348] T 7950b31cbf95472780fa872439374c9a P 854c12bbec8949f397347e2dc7604940 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854c12bbec8949f397347e2dc7604940" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46373 } } peers { permanent_uuid: "8737bbdc600e4cfbaa1fbc71a606db0f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43481 } } peers { permanent_uuid: "48ffd15f6b16463ebd049ab624d11205" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 45517 } }
I20260621 07:50:20.613370 16783 raft_consensus.cc:729] T 7950b31cbf95472780fa872439374c9a P 854c12bbec8949f397347e2dc7604940 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 854c12bbec8949f397347e2dc7604940, State: Initialized, Role: FOLLOWER
I20260621 07:50:20.613791 16783 consensus_queue.cc:260] T 7950b31cbf95472780fa872439374c9a P 854c12bbec8949f397347e2dc7604940 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854c12bbec8949f397347e2dc7604940" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46373 } } peers { permanent_uuid: "8737bbdc600e4cfbaa1fbc71a606db0f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43481 } } peers { permanent_uuid: "48ffd15f6b16463ebd049ab624d11205" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 45517 } }
I20260621 07:50:20.614100 16894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:20.596835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:41431 (local address 127.0.192.254:45149)
0621 07:50:20.597878 (+ 1043us) server_negotiation.cc:206] Beginning negotiation
0621 07:50:20.597896 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0621 07:50:20.597935 (+ 39us) server_negotiation.cc:407] Connection header received
0621 07:50:20.598947 (+ 1012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:50:20.598964 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:50:20.599335 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:50:20.599688 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:50:20.601146 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:20.602769 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:20.604323 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:20.605090 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:20.605289 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:20.607974 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:50:20.607980 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:50:20.607992 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:50:20.608016 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:50:20.609793 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:50:20.610330 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:50:20.610335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:50:20.610339 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:50:20.610467 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:50:20.610839 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:50:20.610842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:50:20.610844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:50:20.611219 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:50:20.612697 (+ 1478us) server_negotiation.cc:1035] Waiting for connection context
0621 07:50:20.612822 (+ 125us) server_negotiation.cc:299] Negotiation successful
0621 07:50:20.613403 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":906,"thread_start_us":96,"threads_started":1}
I20260621 07:50:20.614614 16783 ts_tablet_manager.cc:1428] T 7950b31cbf95472780fa872439374c9a P 854c12bbec8949f397347e2dc7604940: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260621 07:50:20.615964 16891 heartbeater.cc:347] Connected to a master server at 127.0.192.254:45149
I20260621 07:50:20.616181 16891 heartbeater.cc:464] Registering TS with master...
I20260621 07:50:20.616631 16891 heartbeater.cc:511] Master 127.0.192.254:45149 requested a full tablet report, sending...
I20260621 07:50:20.618692 16703 ts_manager.cc:194] Registered new tserver with Master: 854c12bbec8949f397347e2dc7604940 (127.0.192.193:46373)
I20260621 07:50:20.622206 16703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.193@KRBTEST.COM'} at 127.0.192.193:41431
I20260621 07:50:20.624291 16891 heartbeater.cc:503] Master 127.0.192.254:45149 was elected leader, sending a full tablet report...
Jun 21 07:50:20 dist-test-slave-lzx3 krb5kdc[16122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for kudu/127.0.192.193@KRBTEST.COM
I20260621 07:50:20.631063 16898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:20.621626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:46373 (local address 127.0.0.1:60418)
0621 07:50:20.622218 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:20.622247 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0621 07:50:20.622428 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:50:20.622917 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:50:20.622921 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:50:20.623286 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:50:20.623384 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:20.623390 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:20.624733 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:20.624737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:20.625343 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:20.625351 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:20.626022 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:20.626025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:20.626127 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:20.626558 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:50:20.626568 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:50:20.627866 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:50:20.629642 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:50:20.629649 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:50:20.629652 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:50:20.629840 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:50:20.630102 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:50:20.630103 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:50:20.630104 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:50:20.630166 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:50:20.630647 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:50:20.630651 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:50:20.630734 (+ 83us) client_negotiation.cc:769] Sending connection context
0621 07:50:20.630832 (+ 98us) client_negotiation.cc:240] Negotiation successful
0621 07:50:20.630949 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":99,"threads_started":1}
I20260621 07:50:20.631235 16899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:20.621668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60418 (local address 127.0.192.193:46373)
0621 07:50:20.622010 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0621 07:50:20.622020 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:50:20.622336 (+ 316us) server_negotiation.cc:407] Connection header received
0621 07:50:20.622563 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:50:20.622567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:50:20.622690 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:50:20.622814 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:50:20.623496 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:20.624623 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:20.625451 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:20.625894 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:20.626020 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:20.627989 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:50:20.627997 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:50:20.628007 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:50:20.628029 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:50:20.629549 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:50:20.629932 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:50:20.629939 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:50:20.629941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:50:20.630022 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:50:20.630242 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:50:20.630245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:50:20.630246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:50:20.630513 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:50:20.630667 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0621 07:50:20.630943 (+ 276us) server_negotiation.cc:299] Negotiation successful
0621 07:50:20.631090 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":84,"threads_started":1}
I20260621 07:50:20.632014 16825 tablet_service.cc:1514] Processing DeleteTablet for tablet 7950b31cbf95472780fa872439374c9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:50:00 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:60418
I20260621 07:50:20.632783 16900 tablet_replica.cc:330] stopping tablet replica
I20260621 07:50:20.633010 16900 raft_consensus.cc:2229] T 7950b31cbf95472780fa872439374c9a P 854c12bbec8949f397347e2dc7604940 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:50:20.633217 16900 raft_consensus.cc:2258] T 7950b31cbf95472780fa872439374c9a P 854c12bbec8949f397347e2dc7604940 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:50:20.633960 16900 ts_tablet_manager.cc:1905] T 7950b31cbf95472780fa872439374c9a P 854c12bbec8949f397347e2dc7604940: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:50:20.636933 16900 ts_tablet_manager.cc:1918] T 7950b31cbf95472780fa872439374c9a P 854c12bbec8949f397347e2dc7604940: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:50:20.637033 16900 log.cc:1196] T 7950b31cbf95472780fa872439374c9a P 854c12bbec8949f397347e2dc7604940: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/wal/wals/7950b31cbf95472780fa872439374c9a
I20260621 07:50:20.637262 16900 ts_tablet_manager.cc:1939] T 7950b31cbf95472780fa872439374c9a P 854c12bbec8949f397347e2dc7604940: Deleting consensus metadata
I20260621 07:50:20.638480 16687 catalog_manager.cc:4841] TS 854c12bbec8949f397347e2dc7604940 (127.0.192.193:46373): tablet 7950b31cbf95472780fa872439374c9a (table test-table [id=fd374edb15834fe39fb5ea8875535ecd]) successfully deleted
W20260621 07:50:20.704201 16893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:50:20.704407 16893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:50:20.704478 16893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:50:20.708257 16893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:50:20.708300 16893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:50:20.708397 16893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.194
I20260621 07:50:20.713095 16893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:35349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.194:43481
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.192.194
--webserver_port=44757
--tserver_master_addrs=127.0.192.254:45149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.194
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:50:20.713896 16893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:50:20.714772 16893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:50:20.722869 16906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:50:20.722875 16905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:50:20.723044 16893 server_base.cc:1034] running on GCE node
W20260621 07:50:20.723138 16908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:50:20.723489 16893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:50:20.724038 16893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:50:20.725214 16893 hybrid_clock.cc:648] HybridClock initialized: now 1782028220725192 us; error 41 us; skew 500 ppm
Jun 21 07:50:20 dist-test-slave-lzx3 krb5kdc[16122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:50:20.732249 16893 init.cc:377] Logged in from keytab as kudu/127.0.192.194@KRBTEST.COM (short username kudu)
I20260621 07:50:20.733045 16893 webserver.cc:460] Webserver started at http://127.0.192.194:44757/ using document root <none> and password file <none>
I20260621 07:50:20.733558 16893 fs_manager.cc:359] Metadata directory not provided
I20260621 07:50:20.733657 16893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:50:20.737041 16893 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260621 07:50:20.738715 16915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:50:20.739764 16893 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:50:20.739869 16893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/wal
uuid: "8737bbdc600e4cfbaa1fbc71a606db0f"
format_stamp: "Formatted at 2026-06-21 07:49:59 on dist-test-slave-lzx3"
server_key: "f9fd2a9d6bee71d733687e4d29c840b2"
server_key_iv: "e5a4a3aa53f03eaa9be9c286129e59fb"
server_key_version: "encryptionkey@0"
I20260621 07:50:20.740343 16893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:50:20.757882 16893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:50:20.760468 16893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:50:20.760679 16893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:50:20.761201 16893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:50:20.761924 16893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:50:20.761962 16893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:50:20.761994 16893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:50:20.762014 16893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:50:20.770351 16893 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.194:43481
I20260621 07:50:20.770458 17028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.194:43481 every 8 connection(s)
I20260621 07:50:20.771353 16893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
Jun 21 07:50:20 dist-test-slave-lzx3 krb5kdc[16122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:50:20.781133 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 16893
I20260621 07:50:20.781558 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.195:45517
--local_ip_for_outbound_sockets=127.0.192.195
--tserver_master_addrs=127.0.192.254:45149
--webserver_port=32853
--webserver_interface=127.0.192.195
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:35349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:50:20.783078 16894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:20.772983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:41783 (local address 127.0.192.254:45149)
0621 07:50:20.773123 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0621 07:50:20.773126 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:50:20.773809 (+ 683us) server_negotiation.cc:407] Connection header received
0621 07:50:20.774926 (+ 1117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:50:20.774929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:50:20.774989 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:50:20.775087 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:50:20.776072 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:20.776605 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:20.777622 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:20.777862 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:20.777966 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:20.780024 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:50:20.780027 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:50:20.780029 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:50:20.780050 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:50:20.781289 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:50:20.781807 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:50:20.781812 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:50:20.781814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:50:20.781887 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:50:20.782175 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:50:20.782178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:50:20.782179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:50:20.782328 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:50:20.782429 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0621 07:50:20.782819 (+ 390us) server_negotiation.cc:299] Negotiation successful
0621 07:50:20.782933 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 07:50:20.783706 17031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:20.773298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.194:41783)
0621 07:50:20.773677 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:20.773712 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 07:50:20.774735 (+ 1023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:50:20.775238 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:50:20.775244 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:50:20.775538 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:50:20.775933 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:20.775945 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:20.776720 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:20.776723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:20.777485 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:20.777494 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:20.777996 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:20.777999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:20.778171 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:20.778590 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:50:20.778604 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:50:20.779871 (+ 1267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:50:20.781397 (+ 1526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:50:20.781403 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:50:20.781415 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:50:20.781675 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:50:20.781973 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:50:20.781975 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:50:20.781976 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:50:20.782083 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:50:20.782421 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:50:20.782427 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:50:20.782617 (+ 190us) client_negotiation.cc:769] Sending connection context
0621 07:50:20.782770 (+ 153us) client_negotiation.cc:240] Negotiation successful
0621 07:50:20.783028 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":106,"threads_started":1}
I20260621 07:50:20.784849 17029 heartbeater.cc:347] Connected to a master server at 127.0.192.254:45149
I20260621 07:50:20.785050 17029 heartbeater.cc:464] Registering TS with master...
I20260621 07:50:20.785434 17029 heartbeater.cc:511] Master 127.0.192.254:45149 requested a full tablet report, sending...
I20260621 07:50:20.786433 16703 ts_manager.cc:194] Registered new tserver with Master: 8737bbdc600e4cfbaa1fbc71a606db0f (127.0.192.194:43481)
I20260621 07:50:20.787250 16703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.194@KRBTEST.COM'} at 127.0.192.194:41783
W20260621 07:50:20.884749 17032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:50:20.884959 17032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:50:20.884992 17032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:50:20.888411 17032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:50:20.888456 17032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:50:20.888504 17032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.195
I20260621 07:50:20.893589 17032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:35349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.195:45517
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.192.195
--webserver_port=32853
--tserver_master_addrs=127.0.192.254:45149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.195
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:50:20.894268 17032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:50:20.895015 17032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:50:20.903548 17038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:50:20.903616 17037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:50:20.903591 17040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:50:20.903724 17032 server_base.cc:1034] running on GCE node
I20260621 07:50:20.904085 17032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:50:20.904798 17032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:50:20.905975 17032 hybrid_clock.cc:648] HybridClock initialized: now 1782028220905946 us; error 25 us; skew 500 ppm
Jun 21 07:50:20 dist-test-slave-lzx3 krb5kdc[16122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:50:20.908221 17032 init.cc:377] Logged in from keytab as kudu/127.0.192.195@KRBTEST.COM (short username kudu)
I20260621 07:50:20.909010 17032 webserver.cc:460] Webserver started at http://127.0.192.195:32853/ using document root <none> and password file <none>
I20260621 07:50:20.909476 17032 fs_manager.cc:359] Metadata directory not provided
I20260621 07:50:20.909576 17032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:50:20.912983 17032 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:50:20.914624 17047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:50:20.915611 17032 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:50:20.915722 17032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/wal
uuid: "48ffd15f6b16463ebd049ab624d11205"
format_stamp: "Formatted at 2026-06-21 07:50:00 on dist-test-slave-lzx3"
server_key: "d194414a0870e7e7d2f427192faade69"
server_key_iv: "b241a51556df352e2bcadbc68b5de72a"
server_key_version: "encryptionkey@0"
I20260621 07:50:20.916170 17032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:50:20.926455 17032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:50:20.929069 17032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:50:20.929253 17032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:50:20.929759 17032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:50:20.930428 17032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:50:20.930491 17032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:50:20.930564 17032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:50:20.930611 17032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:50:20.939147 17032 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.195:45517
I20260621 07:50:20.939167 17160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.195:45517 every 8 connection(s)
I20260621 07:50:20.939903 17032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
I20260621 07:50:20.947335 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 17032
Jun 21 07:50:20 dist-test-slave-lzx3 krb5kdc[16122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:50:20.952023 16894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:20.941492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:47135 (local address 127.0.192.254:45149)
0621 07:50:20.941628 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0621 07:50:20.941633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:50:20.942802 (+ 1169us) server_negotiation.cc:407] Connection header received
0621 07:50:20.943536 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:50:20.943539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:50:20.943659 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:50:20.943744 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:50:20.944799 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:20.945248 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:20.946135 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:20.946312 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:20.946376 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:20.948491 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:50:20.948494 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:50:20.948497 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:50:20.948517 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:50:20.950156 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:50:20.950572 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:50:20.950575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:50:20.950577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:50:20.950672 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:50:20.951028 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:50:20.951030 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:50:20.951031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:50:20.951182 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:50:20.951254 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0621 07:50:20.951781 (+ 527us) server_negotiation.cc:299] Negotiation successful
0621 07:50:20.951875 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 07:50:20.952775 17163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:20.941894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.195:47135)
0621 07:50:20.942679 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:20.942713 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0621 07:50:20.943345 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:50:20.943890 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:50:20.943896 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:50:20.944188 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:50:20.944656 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:20.944669 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:20.945335 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:20.945337 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:20.946049 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:20.946056 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:20.946413 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:20.946415 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:20.946525 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:20.946924 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:50:20.946937 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:50:20.948337 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:50:20.950247 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:50:20.950250 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:50:20.950258 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:50:20.950466 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:50:20.950770 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:50:20.950773 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:50:20.950775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:50:20.950934 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:50:20.951278 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:50:20.951283 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:50:20.951536 (+ 253us) client_negotiation.cc:769] Sending connection context
0621 07:50:20.951739 (+ 203us) client_negotiation.cc:240] Negotiation successful
0621 07:50:20.951948 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":495,"threads_started":1}
I20260621 07:50:20.953840 17161 heartbeater.cc:347] Connected to a master server at 127.0.192.254:45149
I20260621 07:50:20.954051 17161 heartbeater.cc:464] Registering TS with master...
I20260621 07:50:20.954596 17161 heartbeater.cc:511] Master 127.0.192.254:45149 requested a full tablet report, sending...
I20260621 07:50:20.955641 16703 ts_manager.cc:194] Registered new tserver with Master: 48ffd15f6b16463ebd049ab624d11205 (127.0.192.195:45517)
I20260621 07:50:20.956306 16703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.195@KRBTEST.COM'} at 127.0.192.195:47135
I20260621 07:50:20.956542 17164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:20.947976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45856 (local address 127.0.192.254:45149)
0621 07:50:20.948154 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0621 07:50:20.948156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 07:50:20.948360 (+ 204us) server_negotiation.cc:407] Connection header received
0621 07:50:20.948492 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:50:20.948494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:50:20.948528 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:50:20.948628 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:50:20.949978 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:20.950579 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:20.951747 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:20.952009 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:20.952101 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:20.953553 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:50:20.953556 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:50:20.953558 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:50:20.953579 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:50:20.954993 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:50:20.955438 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:50:20.955441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:50:20.955442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:50:20.955521 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:50:20.955805 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:50:20.955808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:50:20.955809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:50:20.955969 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:50:20.956147 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0621 07:50:20.956237 (+ 90us) server_negotiation.cc:299] Negotiation successful
0621 07:50:20.956365 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":129,"thread_start_us":70,"threads_started":1}
I20260621 07:50:20.958058 771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:50:20.962213 17164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:20.958585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45870 (local address 127.0.192.254:45149)
0621 07:50:20.958724 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0621 07:50:20.958728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:50:20.958881 (+ 153us) server_negotiation.cc:407] Connection header received
0621 07:50:20.959074 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:50:20.959076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:50:20.959208 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:50:20.959324 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:50:20.959537 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:20.960001 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:20.960968 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:20.961186 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:20.961292 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:20.961588 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:50:20.961884 (+ 296us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:50:20.961997 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0621 07:50:20.962060 (+ 63us) server_negotiation.cc:299] Negotiation successful
0621 07:50:20.962102 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 07:50:20.962698 16703 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:45870:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:50:20.965806 16703 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:50:20.972174 16825 tablet_service.cc:1467] Processing CreateTablet for tablet 4913ceae6fd745a4b261a9317aedf95e (DEFAULT_TABLE table=test-table [id=8e3df2357f004f38ab2a2320a8632eb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:50:20.972800 16825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4913ceae6fd745a4b261a9317aedf95e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:50:20.975292 16783 tablet_bootstrap.cc:492] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940: Bootstrap starting.
I20260621 07:50:20.975657 16898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:20.971166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:43481 (local address 127.0.0.1:40534)
0621 07:50:20.971495 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:20.971509 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:50:20.971586 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:50:20.972108 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:50:20.972111 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:50:20.972126 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:50:20.972195 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:20.972199 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:20.973606 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:20.973611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:20.974561 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:20.974569 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:20.975306 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:20.975311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:20.975397 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:20.975435 (+ 38us) client_negotiation.cc:769] Sending connection context
0621 07:50:20.975483 (+ 48us) client_negotiation.cc:240] Negotiation successful
0621 07:50:20.975546 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272}
I20260621 07:50:20.976112 16783 tablet_bootstrap.cc:654] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940: Neither blocks nor log segments found. Creating new log.
I20260621 07:50:20.976248 17167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:20.971297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40534 (local address 127.0.192.194:43481)
0621 07:50:20.971617 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0621 07:50:20.971630 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 07:50:20.971665 (+ 35us) server_negotiation.cc:407] Connection header received
0621 07:50:20.971733 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:50:20.971749 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:50:20.971952 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:50:20.972340 (+ 388us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:50:20.972511 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:20.973508 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:20.974671 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:20.975177 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:20.975815 (+ 638us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:20.975952 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0621 07:50:20.976000 (+ 48us) server_negotiation.cc:299] Negotiation successful
0621 07:50:20.976100 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":98,"threads_started":1}
I20260621 07:50:20.976658 17168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:20.971705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.195:45517 (local address 127.0.0.1:55258)
0621 07:50:20.972428 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:20.972441 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:50:20.972525 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:50:20.973113 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:50:20.973116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:50:20.973131 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:50:20.973197 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:20.973202 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:20.974318 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:20.974323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:20.975267 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:20.975278 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:20.976311 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:20.976316 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:20.976400 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:20.976420 (+ 20us) client_negotiation.cc:769] Sending connection context
0621 07:50:20.976477 (+ 57us) client_negotiation.cc:240] Negotiation successful
0621 07:50:20.976536 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":93,"threads_started":1}
I20260621 07:50:20.976859 17169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:20.972103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55258 (local address 127.0.192.195:45517)
0621 07:50:20.972420 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0621 07:50:20.972429 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:50:20.972501 (+ 72us) server_negotiation.cc:407] Connection header received
0621 07:50:20.972649 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:50:20.972654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:50:20.972839 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:50:20.972974 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:50:20.973313 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:20.974199 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:20.975419 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:20.976038 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:20.976094 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:20.976206 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0621 07:50:20.976513 (+ 307us) server_negotiation.cc:299] Negotiation successful
0621 07:50:20.976605 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":125,"threads_started":1}
I20260621 07:50:20.977609 16783 tablet_bootstrap.cc:492] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940: No bootstrap required, opened a new log
I20260621 07:50:20.977687 16783 ts_tablet_manager.cc:1397] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260621 07:50:20.978080 16783 raft_consensus.cc:348] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8737bbdc600e4cfbaa1fbc71a606db0f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43481 } } peers { permanent_uuid: "48ffd15f6b16463ebd049ab624d11205" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 45517 } } peers { permanent_uuid: "854c12bbec8949f397347e2dc7604940" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46373 } }
I20260621 07:50:20.978181 16783 raft_consensus.cc:374] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:50:20.978212 16783 raft_consensus.cc:729] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 854c12bbec8949f397347e2dc7604940, State: Initialized, Role: FOLLOWER
I20260621 07:50:20.978317 16783 consensus_queue.cc:260] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8737bbdc600e4cfbaa1fbc71a606db0f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43481 } } peers { permanent_uuid: "48ffd15f6b16463ebd049ab624d11205" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 45517 } } peers { permanent_uuid: "854c12bbec8949f397347e2dc7604940" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46373 } }
I20260621 07:50:20.978561 16783 ts_tablet_manager.cc:1428] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260621 07:50:20.979323 17095 tablet_service.cc:1467] Processing CreateTablet for tablet 4913ceae6fd745a4b261a9317aedf95e (DEFAULT_TABLE table=test-table [id=8e3df2357f004f38ab2a2320a8632eb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:50:20.979326 16963 tablet_service.cc:1467] Processing CreateTablet for tablet 4913ceae6fd745a4b261a9317aedf95e (DEFAULT_TABLE table=test-table [id=8e3df2357f004f38ab2a2320a8632eb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:50:20.980206 17095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4913ceae6fd745a4b261a9317aedf95e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:50:20.980263 16963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4913ceae6fd745a4b261a9317aedf95e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:50:20.986219 17171 tablet_bootstrap.cc:492] T 4913ceae6fd745a4b261a9317aedf95e P 48ffd15f6b16463ebd049ab624d11205: Bootstrap starting.
I20260621 07:50:20.986855 17172 tablet_bootstrap.cc:492] T 4913ceae6fd745a4b261a9317aedf95e P 8737bbdc600e4cfbaa1fbc71a606db0f: Bootstrap starting.
I20260621 07:50:20.988368 17171 tablet_bootstrap.cc:654] T 4913ceae6fd745a4b261a9317aedf95e P 48ffd15f6b16463ebd049ab624d11205: Neither blocks nor log segments found. Creating new log.
I20260621 07:50:20.988912 17172 tablet_bootstrap.cc:654] T 4913ceae6fd745a4b261a9317aedf95e P 8737bbdc600e4cfbaa1fbc71a606db0f: Neither blocks nor log segments found. Creating new log.
I20260621 07:50:20.989193 17171 log.cc:824] T 4913ceae6fd745a4b261a9317aedf95e P 48ffd15f6b16463ebd049ab624d11205: Log is configured to *not* fsync() on all Append() calls
I20260621 07:50:20.989611 17172 log.cc:824] T 4913ceae6fd745a4b261a9317aedf95e P 8737bbdc600e4cfbaa1fbc71a606db0f: Log is configured to *not* fsync() on all Append() calls
I20260621 07:50:20.990747 17171 tablet_bootstrap.cc:492] T 4913ceae6fd745a4b261a9317aedf95e P 48ffd15f6b16463ebd049ab624d11205: No bootstrap required, opened a new log
I20260621 07:50:20.990962 17171 ts_tablet_manager.cc:1397] T 4913ceae6fd745a4b261a9317aedf95e P 48ffd15f6b16463ebd049ab624d11205: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260621 07:50:20.991233 17172 tablet_bootstrap.cc:492] T 4913ceae6fd745a4b261a9317aedf95e P 8737bbdc600e4cfbaa1fbc71a606db0f: No bootstrap required, opened a new log
I20260621 07:50:20.991420 17172 ts_tablet_manager.cc:1397] T 4913ceae6fd745a4b261a9317aedf95e P 8737bbdc600e4cfbaa1fbc71a606db0f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260621 07:50:20.993947 17171 raft_consensus.cc:348] T 4913ceae6fd745a4b261a9317aedf95e P 48ffd15f6b16463ebd049ab624d11205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8737bbdc600e4cfbaa1fbc71a606db0f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43481 } } peers { permanent_uuid: "48ffd15f6b16463ebd049ab624d11205" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 45517 } } peers { permanent_uuid: "854c12bbec8949f397347e2dc7604940" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46373 } }
I20260621 07:50:20.994200 17171 raft_consensus.cc:374] T 4913ceae6fd745a4b261a9317aedf95e P 48ffd15f6b16463ebd049ab624d11205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:50:20.994262 17171 raft_consensus.cc:729] T 4913ceae6fd745a4b261a9317aedf95e P 48ffd15f6b16463ebd049ab624d11205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48ffd15f6b16463ebd049ab624d11205, State: Initialized, Role: FOLLOWER
I20260621 07:50:20.994163 17172 raft_consensus.cc:348] T 4913ceae6fd745a4b261a9317aedf95e P 8737bbdc600e4cfbaa1fbc71a606db0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8737bbdc600e4cfbaa1fbc71a606db0f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43481 } } peers { permanent_uuid: "48ffd15f6b16463ebd049ab624d11205" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 45517 } } peers { permanent_uuid: "854c12bbec8949f397347e2dc7604940" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46373 } }
I20260621 07:50:20.994403 17172 raft_consensus.cc:374] T 4913ceae6fd745a4b261a9317aedf95e P 8737bbdc600e4cfbaa1fbc71a606db0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:50:20.994457 17172 raft_consensus.cc:729] T 4913ceae6fd745a4b261a9317aedf95e P 8737bbdc600e4cfbaa1fbc71a606db0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8737bbdc600e4cfbaa1fbc71a606db0f, State: Initialized, Role: FOLLOWER
I20260621 07:50:20.994719 17171 consensus_queue.cc:260] T 4913ceae6fd745a4b261a9317aedf95e P 48ffd15f6b16463ebd049ab624d11205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8737bbdc600e4cfbaa1fbc71a606db0f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43481 } } peers { permanent_uuid: "48ffd15f6b16463ebd049ab624d11205" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 45517 } } peers { permanent_uuid: "854c12bbec8949f397347e2dc7604940" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46373 } }
I20260621 07:50:20.994777 17172 consensus_queue.cc:260] T 4913ceae6fd745a4b261a9317aedf95e P 8737bbdc600e4cfbaa1fbc71a606db0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8737bbdc600e4cfbaa1fbc71a606db0f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43481 } } peers { permanent_uuid: "48ffd15f6b16463ebd049ab624d11205" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 45517 } } peers { permanent_uuid: "854c12bbec8949f397347e2dc7604940" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46373 } }
I20260621 07:50:20.995203 17161 heartbeater.cc:503] Master 127.0.192.254:45149 was elected leader, sending a full tablet report...
I20260621 07:50:20.995306 17029 heartbeater.cc:503] Master 127.0.192.254:45149 was elected leader, sending a full tablet report...
I20260621 07:50:20.995548 17171 ts_tablet_manager.cc:1428] T 4913ceae6fd745a4b261a9317aedf95e P 48ffd15f6b16463ebd049ab624d11205: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260621 07:50:20.995628 17172 ts_tablet_manager.cc:1428] T 4913ceae6fd745a4b261a9317aedf95e P 8737bbdc600e4cfbaa1fbc71a606db0f: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260621 07:50:21.022320 17030 tablet.cc:2369] T 4913ceae6fd745a4b261a9317aedf95e P 8737bbdc600e4cfbaa1fbc71a606db0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 07:50:21.097111 16892 tablet.cc:2369] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:50:21.125348 17175 raft_consensus.cc:482] T 4913ceae6fd745a4b261a9317aedf95e P 48ffd15f6b16463ebd049ab624d11205 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:50:21.125514 17175 raft_consensus.cc:504] T 4913ceae6fd745a4b261a9317aedf95e P 48ffd15f6b16463ebd049ab624d11205 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8737bbdc600e4cfbaa1fbc71a606db0f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43481 } } peers { permanent_uuid: "48ffd15f6b16463ebd049ab624d11205" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 45517 } } peers { permanent_uuid: "854c12bbec8949f397347e2dc7604940" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46373 } }
I20260621 07:50:21.126531 17175 leader_election.cc:288] T 4913ceae6fd745a4b261a9317aedf95e P 48ffd15f6b16463ebd049ab624d11205 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8737bbdc600e4cfbaa1fbc71a606db0f (127.0.192.194:43481), 854c12bbec8949f397347e2dc7604940 (127.0.192.193:46373)
I20260621 07:50:21.131057 16899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:21.126838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:39841 (local address 127.0.192.193:46373)
0621 07:50:21.126961 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0621 07:50:21.126965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:50:21.127159 (+ 194us) server_negotiation.cc:407] Connection header received
0621 07:50:21.127210 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:50:21.127213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:50:21.127266 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:50:21.127358 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:50:21.127691 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:21.128399 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:21.129881 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:21.130488 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:21.130599 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:21.130722 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0621 07:50:21.130868 (+ 146us) server_negotiation.cc:299] Negotiation successful
0621 07:50:21.130939 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260621 07:50:21.131057 17167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:21.126764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:55749 (local address 127.0.192.194:43481)
0621 07:50:21.126878 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0621 07:50:21.126885 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:50:21.127072 (+ 187us) server_negotiation.cc:407] Connection header received
0621 07:50:21.127209 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:50:21.127212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:50:21.127259 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:50:21.127343 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:50:21.127691 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:21.128393 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:21.129881 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:21.130415 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:21.130572 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:21.130622 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0621 07:50:21.130875 (+ 253us) server_negotiation.cc:299] Negotiation successful
0621 07:50:21.130940 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260621 07:50:21.131094 17163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:21.126677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:43481 (local address 127.0.192.195:55749)
0621 07:50:21.126990 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:21.127004 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:50:21.127096 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:50:21.127373 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:50:21.127376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:50:21.127410 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:50:21.127583 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:21.127590 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:21.128494 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:21.128497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:21.129764 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:21.129773 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:21.130536 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:21.130559 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:21.130713 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:21.130761 (+ 48us) client_negotiation.cc:769] Sending connection context
0621 07:50:21.130835 (+ 74us) client_negotiation.cc:240] Negotiation successful
0621 07:50:21.130947 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":158,"spinlock_wait_cycles":25216}
I20260621 07:50:21.131286 17177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:21.126752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:46373 (local address 127.0.192.195:39841)
0621 07:50:21.127061 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:21.127072 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 07:50:21.127150 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:50:21.127377 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:50:21.127379 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:50:21.127412 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:50:21.127588 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:21.127592 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:21.128494 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:21.128497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:21.129764 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:21.129773 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:21.130597 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:21.130600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:21.130703 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:21.130754 (+ 51us) client_negotiation.cc:769] Sending connection context
0621 07:50:21.130838 (+ 84us) client_negotiation.cc:240] Negotiation successful
0621 07:50:21.130947 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":189,"spinlock_wait_cycles":31360,"thread_start_us":121,"threads_started":1}
I20260621 07:50:21.131713 16983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4913ceae6fd745a4b261a9317aedf95e" candidate_uuid: "48ffd15f6b16463ebd049ab624d11205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8737bbdc600e4cfbaa1fbc71a606db0f" is_pre_election: true
I20260621 07:50:21.131861 16845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4913ceae6fd745a4b261a9317aedf95e" candidate_uuid: "48ffd15f6b16463ebd049ab624d11205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "854c12bbec8949f397347e2dc7604940" is_pre_election: true
I20260621 07:50:21.131978 16983 raft_consensus.cc:2454] T 4913ceae6fd745a4b261a9317aedf95e P 8737bbdc600e4cfbaa1fbc71a606db0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48ffd15f6b16463ebd049ab624d11205 in term 0.
I20260621 07:50:21.132083 16845 raft_consensus.cc:2454] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48ffd15f6b16463ebd049ab624d11205 in term 0.
I20260621 07:50:21.132411 17050 leader_election.cc:302] T 4913ceae6fd745a4b261a9317aedf95e P 48ffd15f6b16463ebd049ab624d11205 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 48ffd15f6b16463ebd049ab624d11205, 854c12bbec8949f397347e2dc7604940, 8737bbdc600e4cfbaa1fbc71a606db0f; no voters:
I20260621 07:50:21.132659 17175 raft_consensus.cc:2781] T 4913ceae6fd745a4b261a9317aedf95e P 48ffd15f6b16463ebd049ab624d11205 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:50:21.132743 17175 raft_consensus.cc:482] T 4913ceae6fd745a4b261a9317aedf95e P 48ffd15f6b16463ebd049ab624d11205 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:50:21.132798 17175 raft_consensus.cc:3037] T 4913ceae6fd745a4b261a9317aedf95e P 48ffd15f6b16463ebd049ab624d11205 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:50:21.133755 17175 raft_consensus.cc:504] T 4913ceae6fd745a4b261a9317aedf95e P 48ffd15f6b16463ebd049ab624d11205 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8737bbdc600e4cfbaa1fbc71a606db0f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43481 } } peers { permanent_uuid: "48ffd15f6b16463ebd049ab624d11205" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 45517 } } peers { permanent_uuid: "854c12bbec8949f397347e2dc7604940" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46373 } }
I20260621 07:50:21.134057 17175 leader_election.cc:288] T 4913ceae6fd745a4b261a9317aedf95e P 48ffd15f6b16463ebd049ab624d11205 [CANDIDATE]: Term 1 election: Requested vote from peers 8737bbdc600e4cfbaa1fbc71a606db0f (127.0.192.194:43481), 854c12bbec8949f397347e2dc7604940 (127.0.192.193:46373)
I20260621 07:50:21.134361 16845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4913ceae6fd745a4b261a9317aedf95e" candidate_uuid: "48ffd15f6b16463ebd049ab624d11205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "854c12bbec8949f397347e2dc7604940"
I20260621 07:50:21.134361 16983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4913ceae6fd745a4b261a9317aedf95e" candidate_uuid: "48ffd15f6b16463ebd049ab624d11205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8737bbdc600e4cfbaa1fbc71a606db0f"
I20260621 07:50:21.134467 16845 raft_consensus.cc:3037] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:50:21.134467 16983 raft_consensus.cc:3037] T 4913ceae6fd745a4b261a9317aedf95e P 8737bbdc600e4cfbaa1fbc71a606db0f [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:50:21.135574 16845 raft_consensus.cc:2454] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48ffd15f6b16463ebd049ab624d11205 in term 1.
I20260621 07:50:21.135574 16983 raft_consensus.cc:2454] T 4913ceae6fd745a4b261a9317aedf95e P 8737bbdc600e4cfbaa1fbc71a606db0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48ffd15f6b16463ebd049ab624d11205 in term 1.
I20260621 07:50:21.135829 17048 leader_election.cc:302] T 4913ceae6fd745a4b261a9317aedf95e P 48ffd15f6b16463ebd049ab624d11205 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 48ffd15f6b16463ebd049ab624d11205, 854c12bbec8949f397347e2dc7604940, 8737bbdc600e4cfbaa1fbc71a606db0f; no voters:
I20260621 07:50:21.136003 17175 raft_consensus.cc:2781] T 4913ceae6fd745a4b261a9317aedf95e P 48ffd15f6b16463ebd049ab624d11205 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:50:21.136137 17175 raft_consensus.cc:686] T 4913ceae6fd745a4b261a9317aedf95e P 48ffd15f6b16463ebd049ab624d11205 [term 1 LEADER]: Becoming Leader. State: Replica: 48ffd15f6b16463ebd049ab624d11205, State: Running, Role: LEADER
I20260621 07:50:21.136428 17175 consensus_queue.cc:237] T 4913ceae6fd745a4b261a9317aedf95e P 48ffd15f6b16463ebd049ab624d11205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8737bbdc600e4cfbaa1fbc71a606db0f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43481 } } peers { permanent_uuid: "48ffd15f6b16463ebd049ab624d11205" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 45517 } } peers { permanent_uuid: "854c12bbec8949f397347e2dc7604940" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46373 } }
I20260621 07:50:21.139140 16703 catalog_manager.cc:5495] T 4913ceae6fd745a4b261a9317aedf95e P 48ffd15f6b16463ebd049ab624d11205 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48ffd15f6b16463ebd049ab624d11205 (127.0.192.195). New cstate: current_term: 1 leader_uuid: "48ffd15f6b16463ebd049ab624d11205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8737bbdc600e4cfbaa1fbc71a606db0f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48ffd15f6b16463ebd049ab624d11205" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 45517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "854c12bbec8949f397347e2dc7604940" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46373 } health_report { overall_health: UNKNOWN } } }
I20260621 07:50:21.162410 17169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:21.159447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55260 (local address 127.0.192.195:45517)
0621 07:50:21.159550 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0621 07:50:21.159553 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:50:21.159566 (+ 13us) server_negotiation.cc:407] Connection header received
0621 07:50:21.159749 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:50:21.159752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:50:21.159829 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:50:21.159928 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:50:21.160175 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:21.160713 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:21.161521 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:21.161732 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:21.161807 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:21.162029 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:50:21.162145 (+ 116us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0621 07:50:21.162320 (+ 175us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55260: token expired
Metrics: {"server-negotiator.queue_time_us":40}
W20260621 07:50:21.162484 17169 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55260: token expired
I20260621 07:50:21.169575 17164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:21.163109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45876 (local address 127.0.192.254:45149)
0621 07:50:21.163235 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0621 07:50:21.163239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:50:21.163250 (+ 11us) server_negotiation.cc:407] Connection header received
0621 07:50:21.163363 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:50:21.163366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:50:21.163400 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:50:21.163476 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:50:21.164119 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:21.164680 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:21.165460 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:21.165662 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:21.165736 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:21.166852 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:50:21.166855 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:50:21.166856 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:50:21.166895 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:50:21.168350 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:50:21.168758 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:50:21.168760 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:50:21.168761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:50:21.168828 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:50:21.169020 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:50:21.169021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:50:21.169022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:50:21.169152 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:50:21.169239 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0621 07:50:21.169390 (+ 151us) server_negotiation.cc:299] Negotiation successful
0621 07:50:21.169475 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260621 07:50:21.173615 17169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:21.170963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55270 (local address 127.0.192.195:45517)
0621 07:50:21.171070 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0621 07:50:21.171074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:50:21.171087 (+ 13us) server_negotiation.cc:407] Connection header received
0621 07:50:21.171178 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:50:21.171180 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:50:21.171216 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:50:21.171276 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:50:21.171440 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:21.171887 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:21.172540 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:21.172788 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:21.172867 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:21.173044 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:50:21.173342 (+ 298us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:50:21.173402 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0621 07:50:21.173493 (+ 91us) server_negotiation.cc:299] Negotiation successful
0621 07:50:21.173533 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260621 07:50:21.179086 16845 raft_consensus.cc:1261] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [term 1 FOLLOWER]: Refusing update from remote peer 48ffd15f6b16463ebd049ab624d11205: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:50:21.179091 16983 raft_consensus.cc:1261] T 4913ceae6fd745a4b261a9317aedf95e P 8737bbdc600e4cfbaa1fbc71a606db0f [term 1 FOLLOWER]: Refusing update from remote peer 48ffd15f6b16463ebd049ab624d11205: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:50:21.179778 17175 consensus_queue.cc:1035] T 4913ceae6fd745a4b261a9317aedf95e P 48ffd15f6b16463ebd049ab624d11205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "854c12bbec8949f397347e2dc7604940" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:50:21.179975 17178 consensus_queue.cc:1035] T 4913ceae6fd745a4b261a9317aedf95e P 48ffd15f6b16463ebd049ab624d11205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8737bbdc600e4cfbaa1fbc71a606db0f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:50:21.185279 17181 mvcc.cc:205] Tried to move back new op lower bound from 7299187593942982656 to 7299187593777520640. Current Snapshot: MvccSnapshot[applied={T|T < 7299187593942982656}]
I20260621 07:50:21.189543 17184 mvcc.cc:205] Tried to move back new op lower bound from 7299187593942982656 to 7299187593777520640. Current Snapshot: MvccSnapshot[applied={T|T < 7299187593942982656}]
I20260621 07:50:21.194363 16701 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:50:21.194510 16701 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:50:21.196460 16701 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c: Sending DeleteTablet for 3 replicas of tablet 4913ceae6fd745a4b261a9317aedf95e
I20260621 07:50:21.197043 16963 tablet_service.cc:1514] Processing DeleteTablet for tablet 4913ceae6fd745a4b261a9317aedf95e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:50:21 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:40534
I20260621 07:50:21.197147 17095 tablet_service.cc:1514] Processing DeleteTablet for tablet 4913ceae6fd745a4b261a9317aedf95e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:50:21 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:55258
I20260621 07:50:21.197204 16825 tablet_service.cc:1514] Processing DeleteTablet for tablet 4913ceae6fd745a4b261a9317aedf95e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:50:21 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:60418
I20260621 07:50:21.197570 17190 tablet_replica.cc:330] T 4913ceae6fd745a4b261a9317aedf95e P 8737bbdc600e4cfbaa1fbc71a606db0f: stopping tablet replica
I20260621 07:50:21.197579 17191 tablet_replica.cc:330] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940: stopping tablet replica
I20260621 07:50:21.197602 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 16741
I20260621 07:50:21.198312 17192 tablet_replica.cc:330] T 4913ceae6fd745a4b261a9317aedf95e P 48ffd15f6b16463ebd049ab624d11205: stopping tablet replica
I20260621 07:50:21.198455 17190 raft_consensus.cc:2229] T 4913ceae6fd745a4b261a9317aedf95e P 8737bbdc600e4cfbaa1fbc71a606db0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:50:21.198571 17192 raft_consensus.cc:2229] T 4913ceae6fd745a4b261a9317aedf95e P 48ffd15f6b16463ebd049ab624d11205 [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:50:21.198732 17190 raft_consensus.cc:2258] T 4913ceae6fd745a4b261a9317aedf95e P 8737bbdc600e4cfbaa1fbc71a606db0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:50:21.198894 17192 raft_consensus.cc:2258] T 4913ceae6fd745a4b261a9317aedf95e P 48ffd15f6b16463ebd049ab624d11205 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:50:21.200116 17190 ts_tablet_manager.cc:1905] T 4913ceae6fd745a4b261a9317aedf95e P 8737bbdc600e4cfbaa1fbc71a606db0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:50:21.200269 17192 ts_tablet_manager.cc:1905] T 4913ceae6fd745a4b261a9317aedf95e P 48ffd15f6b16463ebd049ab624d11205: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:50:21.202931 17190 ts_tablet_manager.cc:1918] T 4913ceae6fd745a4b261a9317aedf95e P 8737bbdc600e4cfbaa1fbc71a606db0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:50:21.203078 17190 log.cc:1196] T 4913ceae6fd745a4b261a9317aedf95e P 8737bbdc600e4cfbaa1fbc71a606db0f: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/wal/wals/4913ceae6fd745a4b261a9317aedf95e
I20260621 07:50:21.203512 17190 ts_tablet_manager.cc:1939] T 4913ceae6fd745a4b261a9317aedf95e P 8737bbdc600e4cfbaa1fbc71a606db0f: Deleting consensus metadata
I20260621 07:50:21.203593 17192 ts_tablet_manager.cc:1918] T 4913ceae6fd745a4b261a9317aedf95e P 48ffd15f6b16463ebd049ab624d11205: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:50:21.203691 17192 log.cc:1196] T 4913ceae6fd745a4b261a9317aedf95e P 48ffd15f6b16463ebd049ab624d11205: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/wal/wals/4913ceae6fd745a4b261a9317aedf95e
I20260621 07:50:21.204039 17192 ts_tablet_manager.cc:1939] T 4913ceae6fd745a4b261a9317aedf95e P 48ffd15f6b16463ebd049ab624d11205: Deleting consensus metadata
I20260621 07:50:21.204887 16689 catalog_manager.cc:4841] TS 8737bbdc600e4cfbaa1fbc71a606db0f (127.0.192.194:43481): tablet 4913ceae6fd745a4b261a9317aedf95e (table test-table [id=8e3df2357f004f38ab2a2320a8632eb6]) successfully deleted
I20260621 07:50:21.205181 16688 catalog_manager.cc:4841] TS 48ffd15f6b16463ebd049ab624d11205 (127.0.192.195:45517): tablet 4913ceae6fd745a4b261a9317aedf95e (table test-table [id=8e3df2357f004f38ab2a2320a8632eb6]) successfully deleted
W20260621 07:50:21.206645 16687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.192.193:46373 (error 108)
I20260621 07:50:21.207034 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 16893
I20260621 07:50:21.209292 17168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:21.209030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:46373 (local address 127.0.0.1:60426)
0621 07:50:21.209151 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:21.209225 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.193:46373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260621 07:50:21.209460 16687 catalog_manager.cc:4584] TS 854c12bbec8949f397347e2dc7604940 (127.0.192.193:46373): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4913ceae6fd745a4b261a9317aedf95e: Network error: Client connection negotiation failed: client connection to 127.0.192.193:46373: connect: Connection refused (error 111)
I20260621 07:50:21.214881 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 17032
I20260621 07:50:21.220605 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 16670
I20260621 07:50:41.226912 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:45149
--webserver_interface=127.0.192.254
--webserver_port=40327
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:35349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:45149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:50:41.325922 17194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:50:41.326143 17194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:50:41.326179 17194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:50:41.329536 17194 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260621 07:50:41.329571 17194 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260621 07:50:41.329582 17194 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260621 07:50:41.329594 17194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:50:41.329608 17194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:50:41.329627 17194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:50:41.329641 17194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:50:41.329654 17194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:50:41.333823 17194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:35349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:45149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:45149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=40327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:50:41.334511 17194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:50:41.335222 17194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:50:41.342869 17202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:50:41.342897 17194 server_base.cc:1034] running on GCE node
W20260621 07:50:41.342865 17199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:50:41.342887 17200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:50:41.343713 17194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:50:41.344642 17194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:50:41.345817 17194 hybrid_clock.cc:648] HybridClock initialized: now 1782028241345796 us; error 35 us; skew 500 ppm
Jun 21 07:50:41 dist-test-slave-lzx3 krb5kdc[16122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:50:41.351811 17194 init.cc:377] Logged in from keytab as kudu/127.0.192.254@KRBTEST.COM (short username kudu)
I20260621 07:50:41.352535 17194 webserver.cc:460] Webserver started at http://127.0.192.254:40327/ using document root <none> and password file <none>
I20260621 07:50:41.353004 17194 fs_manager.cc:359] Metadata directory not provided
I20260621 07:50:41.353060 17194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:50:41.356276 17194 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260621 07:50:41.358237 17209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:50:41.359313 17194 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 07:50:41.359407 17194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "98c78c73eb2247848e49787843c20f5c"
format_stamp: "Formatted at 2026-06-21 07:49:59 on dist-test-slave-lzx3"
server_key: "7f1307aaa489feea232904a5e07855ce"
server_key_iv: "a66aad73380eec7c6c8f0531051c12c0"
server_key_version: "encryptionkey@0"
I20260621 07:50:41.359824 17194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:50:41.369493 17194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:50:41.372213 17194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:50:41.372350 17194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:50:41.379050 17194 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:45149
I20260621 07:50:41.379298 17261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:45149 every 8 connection(s)
I20260621 07:50:41.380010 17194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:50:41.382198 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 17194
I20260621 07:50:41.382603 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.193:46373
--local_ip_for_outbound_sockets=127.0.192.193
--tserver_master_addrs=127.0.192.254:45149
--webserver_port=42439
--webserver_interface=127.0.192.193
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:35349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:50:41.383661 17262 sys_catalog.cc:263] Verifying existing consensus state
I20260621 07:50:41.385183 17262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c: Bootstrap starting.
I20260621 07:50:41.402191 17262 log.cc:824] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c: Log is configured to *not* fsync() on all Append() calls
I20260621 07:50:41.411468 17262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 07:50:41.411791 17262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c: Bootstrap complete.
I20260621 07:50:41.415207 17262 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c78c73eb2247848e49787843c20f5c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45149 } }
I20260621 07:50:41.415549 17262 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98c78c73eb2247848e49787843c20f5c, State: Initialized, Role: FOLLOWER
I20260621 07:50:41.415951 17262 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c78c73eb2247848e49787843c20f5c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45149 } }
I20260621 07:50:41.416106 17262 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:50:41.416182 17262 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:50:41.416291 17262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [term 2 FOLLOWER]: Advancing to term 3
I20260621 07:50:41.418035 17262 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c78c73eb2247848e49787843c20f5c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45149 } }
I20260621 07:50:41.418345 17262 leader_election.cc:302] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98c78c73eb2247848e49787843c20f5c; no voters:
I20260621 07:50:41.418642 17262 leader_election.cc:288] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260621 07:50:41.418706 17266 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [term 3 FOLLOWER]: Leader election won for term 3
I20260621 07:50:41.418900 17266 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [term 3 LEADER]: Becoming Leader. State: Replica: 98c78c73eb2247848e49787843c20f5c, State: Running, Role: LEADER
I20260621 07:50:41.419203 17266 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c78c73eb2247848e49787843c20f5c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45149 } }
I20260621 07:50:41.419689 17262 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:50:41.420004 17266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98c78c73eb2247848e49787843c20f5c. Latest consensus state: current_term: 3 leader_uuid: "98c78c73eb2247848e49787843c20f5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c78c73eb2247848e49787843c20f5c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45149 } } }
I20260621 07:50:41.420167 17266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [sys.catalog]: This master's current role is: LEADER
I20260621 07:50:41.420320 17267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "98c78c73eb2247848e49787843c20f5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c78c73eb2247848e49787843c20f5c" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45149 } } }
I20260621 07:50:41.420488 17267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c [sys.catalog]: This master's current role is: LEADER
I20260621 07:50:41.420580 17270 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:50:41.425156 17270 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:50:41.425732 17270 catalog_manager.cc:1259] Loaded cluster ID: 6831487686084c91b5de07c168ebca26
I20260621 07:50:41.425788 17270 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:50:41.428126 17270 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:50:41.428854 17270 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c: Loaded TSK: 0
I20260621 07:50:41.428928 17270 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 98c78c73eb2247848e49787843c20f5c: Loaded TSK: 1
F20260621 07:50:41.429579 17270 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f14f8abe9ad google::LogMessage::Fail() at ??:0
@ 0x7f14f8ac188a google::LogMessage::SendToLog() at ??:0
@ 0x7f14f8abe4de google::LogMessage::Flush() at ??:0
@ 0x7f14f8abfa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f14fb78218d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f14fb77af0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f15034277bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f150340199c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f1503402178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f1503443df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f150346282b std::function<>::operator()() at ??:0
@ 0x7f1503401b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f1503402f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f15033ff579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f1503443640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f14fad103d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f14fad10c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f14fad123f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f14fad01512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f14fc3456db start_thread at ??:0
@ 0x7f14f75bc61f clone at ??:0
W20260621 07:50:41.500319 17264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:50:41.500501 17264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:50:41.500536 17264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:50:41.504643 17264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:50:41.504678 17264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:50:41.504724 17264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.193
I20260621 07:50:41.510794 17264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:35349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.193:46373
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.192.193
--webserver_port=42439
--tserver_master_addrs=127.0.192.254:45149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.193
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:50:41.511940 17264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:50:41.513106 17264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:50:41.524763 17308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:50:41.524809 17310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:50:41.524745 17307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:50:41.525142 17264 server_base.cc:1034] running on GCE node
I20260621 07:50:41.525457 17264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:50:41.526010 17264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:50:41.527202 17264 hybrid_clock.cc:648] HybridClock initialized: now 1782028241527185 us; error 32 us; skew 500 ppm
Jun 21 07:50:41 dist-test-slave-lzx3 krb5kdc[16122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:50:41.530164 17264 init.cc:377] Logged in from keytab as kudu/127.0.192.193@KRBTEST.COM (short username kudu)
I20260621 07:50:41.530889 17264 webserver.cc:460] Webserver started at http://127.0.192.193:42439/ using document root <none> and password file <none>
I20260621 07:50:41.531368 17264 fs_manager.cc:359] Metadata directory not provided
I20260621 07:50:41.531455 17264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:50:41.534907 17264 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260621 07:50:41.536913 17319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:50:41.538106 17264 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260621 07:50:41.538229 17264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/wal
uuid: "854c12bbec8949f397347e2dc7604940"
format_stamp: "Formatted at 2026-06-21 07:49:59 on dist-test-slave-lzx3"
server_key: "c7258f611355283400ad1fb45aee31a1"
server_key_iv: "0585450526b4f4e32e317a48b8232c9c"
server_key_version: "encryptionkey@0"
I20260621 07:50:41.538708 17264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:50:41.547225 17264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:50:41.549876 17264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:50:41.550073 17264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:50:41.550571 17264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:50:41.551546 17326 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260621 07:50:41.556209 17264 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260621 07:50:41.556277 17264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260621 07:50:41.556363 17264 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260621 07:50:41.557886 17264 ts_tablet_manager.cc:610] Registered 1 tablets
I20260621 07:50:41.557963 17264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260621 07:50:41.558202 17326 tablet_bootstrap.cc:492] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940: Bootstrap starting.
I20260621 07:50:41.567185 17264 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.193:46373
I20260621 07:50:41.567345 17433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.193:46373 every 8 connection(s)
I20260621 07:50:41.568053 17264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
I20260621 07:50:41.568311 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 17264
I20260621 07:50:41.568876 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.194:43481
--local_ip_for_outbound_sockets=127.0.192.194
--tserver_master_addrs=127.0.192.254:45149
--webserver_port=44757
--webserver_interface=127.0.192.194
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:35349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:50:41.578979 17326 log.cc:824] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940: Log is configured to *not* fsync() on all Append() calls
I20260621 07:50:41.587005 17326 tablet_bootstrap.cc:492] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 07:50:41.587433 17326 tablet_bootstrap.cc:492] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940: Bootstrap complete.
I20260621 07:50:41.588682 17326 ts_tablet_manager.cc:1397] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940: Time spent bootstrapping tablet: real 0.031s user 0.021s sys 0.007s
I20260621 07:50:41.591563 17326 raft_consensus.cc:348] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8737bbdc600e4cfbaa1fbc71a606db0f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43481 } } peers { permanent_uuid: "48ffd15f6b16463ebd049ab624d11205" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 45517 } } peers { permanent_uuid: "854c12bbec8949f397347e2dc7604940" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46373 } }
I20260621 07:50:41.592042 17326 raft_consensus.cc:729] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 854c12bbec8949f397347e2dc7604940, State: Initialized, Role: FOLLOWER
I20260621 07:50:41.592528 17326 consensus_queue.cc:260] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8737bbdc600e4cfbaa1fbc71a606db0f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43481 } } peers { permanent_uuid: "48ffd15f6b16463ebd049ab624d11205" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 45517 } } peers { permanent_uuid: "854c12bbec8949f397347e2dc7604940" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46373 } }
I20260621 07:50:41.593433 17326 ts_tablet_manager.cc:1428] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260621 07:50:41.644445 17437 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:50:41.570442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.193:48301)
0621 07:50:41.570928 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:41.570983 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0621 07:50:41.571685 (+ 702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:50:41.643815 (+ 72130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":120,"threads_started":1}
W20260621 07:50:41.644927 17434 heartbeater.cc:644] Failed to heartbeat to 127.0.192.254:45149 (0 consecutive failures): Network error: Failed to ping master at 127.0.192.254:45149: Client connection negotiation failed: client connection to 127.0.192.254:45149: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260621 07:50:41.645571 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:41.645363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.193:59857)
0621 07:50:41.645467 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:41.645521 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260621 07:50:41.646239 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:41.646075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.193:45953)
0621 07:50:41.646160 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:41.646198 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260621 07:50:41.646466 17434 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 07:50:41.669279 17436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:50:41.669484 17436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:50:41.669545 17436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:50:41.673796 17436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:50:41.673863 17436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:50:41.673975 17436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.194
I20260621 07:50:41.681025 17436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:35349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.194:43481
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.192.194
--webserver_port=44757
--tserver_master_addrs=127.0.192.254:45149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.194
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:50:41.681779 17436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:50:41.682566 17436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:50:41.690815 17447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:50:41.690842 17445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:50:41.690845 17444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:50:41.690996 17436 server_base.cc:1034] running on GCE node
I20260621 07:50:41.691464 17436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:50:41.692039 17436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:50:41.693181 17436 hybrid_clock.cc:648] HybridClock initialized: now 1782028241693169 us; error 24 us; skew 500 ppm
Jun 21 07:50:41 dist-test-slave-lzx3 krb5kdc[16122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:50:41.695304 17436 init.cc:377] Logged in from keytab as kudu/127.0.192.194@KRBTEST.COM (short username kudu)
I20260621 07:50:41.696089 17436 webserver.cc:460] Webserver started at http://127.0.192.194:44757/ using document root <none> and password file <none>
I20260621 07:50:41.696643 17436 fs_manager.cc:359] Metadata directory not provided
I20260621 07:50:41.696728 17436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:50:41.700230 17436 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260621 07:50:41.702001 17454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:50:41.703043 17436 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 07:50:41.703172 17436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/wal
uuid: "8737bbdc600e4cfbaa1fbc71a606db0f"
format_stamp: "Formatted at 2026-06-21 07:49:59 on dist-test-slave-lzx3"
server_key: "f9fd2a9d6bee71d733687e4d29c840b2"
server_key_iv: "e5a4a3aa53f03eaa9be9c286129e59fb"
server_key_version: "encryptionkey@0"
I20260621 07:50:41.703615 17436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:50:41.726325 17436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:50:41.728911 17436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:50:41.729099 17436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:50:41.729620 17436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:50:41.730355 17436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:50:41.730419 17436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:50:41.730479 17436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:50:41.730520 17436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:50:41.739444 17436 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.194:43481
I20260621 07:50:41.739475 17567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.194:43481 every 8 connection(s)
I20260621 07:50:41.740372 17436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
I20260621 07:50:41.743875 17570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:41.742318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.194:33891)
0621 07:50:41.742770 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:41.743190 (+ 420us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":101,"threads_started":1}
W20260621 07:50:41.744272 17568 heartbeater.cc:644] Failed to heartbeat to 127.0.192.254:45149 (0 consecutive failures): Network error: Failed to ping master at 127.0.192.254:45149: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
I20260621 07:50:41.744848 17570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:41.744657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.194:43765)
0621 07:50:41.744745 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:41.744793 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260621 07:50:41.745160 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 17436
I20260621 07:50:41.745474 17570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:41.745286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.194:40607)
0621 07:50:41.745383 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:41.745429 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260621 07:50:41.745643 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.195:45517
--local_ip_for_outbound_sockets=127.0.192.195
--tserver_master_addrs=127.0.192.254:45149
--webserver_port=32853
--webserver_interface=127.0.192.195
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:35349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:50:41.745673 17568 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 07:50:41.847868 17571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:50:41.848093 17571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:50:41.848133 17571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:50:41.851657 17571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:50:41.851715 17571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:50:41.851780 17571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.195
I20260621 07:50:41.856416 17571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:35349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.195:45517
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.192.195
--webserver_port=32853
--tserver_master_addrs=127.0.192.254:45149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.195
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:50:41.857375 17571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:50:41.858276 17571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:50:41.867126 17579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:50:41.867123 17577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:50:41.867304 17576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:50:41.867370 17571 server_base.cc:1034] running on GCE node
I20260621 07:50:41.867740 17571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:50:41.868283 17571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:50:41.869445 17571 hybrid_clock.cc:648] HybridClock initialized: now 1782028241869423 us; error 38 us; skew 500 ppm
Jun 21 07:50:41 dist-test-slave-lzx3 krb5kdc[16122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:50:41.871786 17571 init.cc:377] Logged in from keytab as kudu/127.0.192.195@KRBTEST.COM (short username kudu)
I20260621 07:50:41.872701 17571 webserver.cc:460] Webserver started at http://127.0.192.195:32853/ using document root <none> and password file <none>
I20260621 07:50:41.873246 17571 fs_manager.cc:359] Metadata directory not provided
I20260621 07:50:41.873325 17571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:50:41.876710 17571 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.003s
I20260621 07:50:41.878607 17586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:50:41.879628 17571 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:50:41.879762 17571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/wal
uuid: "48ffd15f6b16463ebd049ab624d11205"
format_stamp: "Formatted at 2026-06-21 07:50:00 on dist-test-slave-lzx3"
server_key: "d194414a0870e7e7d2f427192faade69"
server_key_iv: "b241a51556df352e2bcadbc68b5de72a"
server_key_version: "encryptionkey@0"
I20260621 07:50:41.880235 17571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:50:41.897167 17571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:50:41.899899 17571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:50:41.900019 17571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:50:41.900452 17571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:50:41.901250 17571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:50:41.901291 17571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:50:41.901324 17571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:50:41.901342 17571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:50:41.908712 17571 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.195:45517
I20260621 07:50:41.908742 17699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.195:45517 every 8 connection(s)
I20260621 07:50:41.909484 17571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
I20260621 07:50:41.911121 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 17571
I20260621 07:50:41.912654 17702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:41.911215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.195:54959)
0621 07:50:41.911619 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:41.912049 (+ 430us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":99,"threads_started":1}
W20260621 07:50:41.913017 17700 heartbeater.cc:644] Failed to heartbeat to 127.0.192.254:45149 (0 consecutive failures): Network error: Failed to ping master at 127.0.192.254:45149: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
I20260621 07:50:41.913475 17702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:41.913316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.195:49889)
0621 07:50:41.913394 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:41.913433 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260621 07:50:41.913980 17702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:41.913839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.195:45241)
0621 07:50:41.913910 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:41.913945 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260621 07:50:41.914155 17700 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260621 07:50:42.647297 17704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:42.646948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.193:44445)
0621 07:50:42.647158 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:42.647201 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":75,"threads_started":1}
I20260621 07:50:42.746512 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:42.746170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.194:55645)
0621 07:50:42.746382 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:42.746428 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":64,"threads_started":1}
I20260621 07:50:42.805956 17706 raft_consensus.cc:482] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:50:42.806170 17706 raft_consensus.cc:504] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8737bbdc600e4cfbaa1fbc71a606db0f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43481 } } peers { permanent_uuid: "48ffd15f6b16463ebd049ab624d11205" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 45517 } } peers { permanent_uuid: "854c12bbec8949f397347e2dc7604940" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46373 } }
I20260621 07:50:42.807361 17706 leader_election.cc:288] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8737bbdc600e4cfbaa1fbc71a606db0f (127.0.192.194:43481), 48ffd15f6b16463ebd049ab624d11205 (127.0.192.195:45517)
Jun 21 07:50:42 dist-test-slave-lzx3 krb5kdc[16122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for kudu/127.0.192.194@KRBTEST.COM
Jun 21 07:50:42 dist-test-slave-lzx3 krb5kdc[16122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for kudu/127.0.192.195@KRBTEST.COM
I20260621 07:50:42.820127 17704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:42.807495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:43481 (local address 127.0.192.193:39627)
0621 07:50:42.807621 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:42.807636 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:50:42.807736 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:50:42.809225 (+ 1489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:50:42.809241 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:50:42.809719 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:50:42.810131 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:42.810142 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:42.811242 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:42.811245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:42.812042 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:42.812051 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:42.812945 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:42.812948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:42.813193 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:42.813827 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:50:42.813847 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:50:42.815569 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:50:42.818095 (+ 2526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:50:42.818099 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:50:42.818108 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:50:42.818344 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:50:42.818702 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:50:42.818704 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:50:42.818705 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:50:42.818829 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:50:42.819389 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:50:42.819391 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:50:42.819555 (+ 164us) client_negotiation.cc:769] Sending connection context
0621 07:50:42.819710 (+ 155us) client_negotiation.cc:240] Negotiation successful
0621 07:50:42.819972 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":193,"spinlock_wait_cycles":12544}
I20260621 07:50:42.820176 17709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:42.807826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:41859 (local address 127.0.192.195:45517)
0621 07:50:42.808138 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0621 07:50:42.808147 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:50:42.808176 (+ 29us) server_negotiation.cc:407] Connection header received
0621 07:50:42.808782 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:50:42.808788 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:50:42.809094 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:50:42.809366 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:50:42.810286 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:42.811118 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:42.812152 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:42.812758 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:42.812909 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:42.816076 (+ 3167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:50:42.816086 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:50:42.816101 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:50:42.816136 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:50:42.817999 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:50:42.818484 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:50:42.818487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:50:42.818491 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:50:42.818582 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:50:42.818911 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:50:42.818916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:50:42.818919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:50:42.819228 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:50:42.819310 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0621 07:50:42.819820 (+ 510us) server_negotiation.cc:299] Negotiation successful
0621 07:50:42.820037 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":88,"threads_started":1}
I20260621 07:50:42.820240 17708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:42.807614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:39627 (local address 127.0.192.194:43481)
0621 07:50:42.807858 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0621 07:50:42.807880 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0621 07:50:42.807918 (+ 38us) server_negotiation.cc:407] Connection header received
0621 07:50:42.808546 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:50:42.808595 (+ 49us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:50:42.808908 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:50:42.809121 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:50:42.810303 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:42.811104 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:42.812174 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:42.812635 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:42.812776 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:42.815830 (+ 3054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:50:42.815835 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:50:42.815844 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:50:42.815871 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:50:42.817990 (+ 2119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:50:42.818447 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:50:42.818455 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:50:42.818461 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:50:42.818580 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:50:42.818935 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:50:42.818938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:50:42.818940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:50:42.819258 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:50:42.819386 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0621 07:50:42.819815 (+ 429us) server_negotiation.cc:299] Negotiation successful
0621 07:50:42.820059 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":60,"threads_started":1}
I20260621 07:50:42.820369 17707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:42.807561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.195:45517 (local address 127.0.192.193:41859)
0621 07:50:42.807920 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:42.807933 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:50:42.808015 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:50:42.809324 (+ 1309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:50:42.809327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:50:42.809710 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:50:42.810129 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:42.810141 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:42.811263 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:42.811266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:42.812039 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:42.812047 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:42.812945 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:42.812948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:42.813192 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:42.813851 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:50:42.813861 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:50:42.815896 (+ 2035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:50:42.818134 (+ 2238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:50:42.818137 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:50:42.818139 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:50:42.818368 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:50:42.818677 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:50:42.818680 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:50:42.818682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:50:42.818823 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:50:42.819347 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:50:42.819353 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:50:42.819562 (+ 209us) client_negotiation.cc:769] Sending connection context
0621 07:50:42.819712 (+ 150us) client_negotiation.cc:240] Negotiation successful
0621 07:50:42.820061 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":271,"spinlock_wait_cycles":35456,"thread_start_us":58,"threads_started":1}
W20260621 07:50:42.822669 17321 leader_election.cc:341] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 48ffd15f6b16463ebd049ab624d11205 (127.0.192.195:45517): Not found: Tablet not found: 4913ceae6fd745a4b261a9317aedf95e
W20260621 07:50:42.822803 17322 leader_election.cc:341] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8737bbdc600e4cfbaa1fbc71a606db0f (127.0.192.194:43481): Not found: Tablet not found: 4913ceae6fd745a4b261a9317aedf95e
I20260621 07:50:42.822903 17322 leader_election.cc:302] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 854c12bbec8949f397347e2dc7604940; no voters: 48ffd15f6b16463ebd049ab624d11205, 8737bbdc600e4cfbaa1fbc71a606db0f
I20260621 07:50:42.823153 17706 raft_consensus.cc:2726] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 07:50:42.914873 17710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:42.914558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.195:51605)
0621 07:50:42.914760 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:42.914802 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":67,"threads_started":1}
I20260621 07:50:43.648452 17711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:43.647970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.193:43039)
0621 07:50:43.648303 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:43.648355 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":114,"threads_started":1}
I20260621 07:50:43.747491 17712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:43.747096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.194:58839)
0621 07:50:43.747350 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:43.747399 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":78,"threads_started":1}
I20260621 07:50:43.915850 17713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:43.915496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.195:48965)
0621 07:50:43.915707 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:43.915756 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":66,"threads_started":1}
I20260621 07:50:44.649428 17714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:44.649095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.193:57567)
0621 07:50:44.649297 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:44.649343 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":62,"threads_started":1}
I20260621 07:50:44.748421 17715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:44.748075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.194:41781)
0621 07:50:44.748285 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:44.748332 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":62,"threads_started":1}
I20260621 07:50:44.897339 17716 raft_consensus.cc:482] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:50:44.897446 17716 raft_consensus.cc:504] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8737bbdc600e4cfbaa1fbc71a606db0f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43481 } } peers { permanent_uuid: "48ffd15f6b16463ebd049ab624d11205" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 45517 } } peers { permanent_uuid: "854c12bbec8949f397347e2dc7604940" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46373 } }
I20260621 07:50:44.897758 17716 leader_election.cc:288] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8737bbdc600e4cfbaa1fbc71a606db0f (127.0.192.194:43481), 48ffd15f6b16463ebd049ab624d11205 (127.0.192.195:45517)
W20260621 07:50:44.898488 17321 leader_election.cc:341] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 48ffd15f6b16463ebd049ab624d11205 (127.0.192.195:45517): Not found: Tablet not found: 4913ceae6fd745a4b261a9317aedf95e
W20260621 07:50:44.898602 17322 leader_election.cc:341] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8737bbdc600e4cfbaa1fbc71a606db0f (127.0.192.194:43481): Not found: Tablet not found: 4913ceae6fd745a4b261a9317aedf95e
I20260621 07:50:44.898655 17322 leader_election.cc:302] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 854c12bbec8949f397347e2dc7604940; no voters: 48ffd15f6b16463ebd049ab624d11205, 8737bbdc600e4cfbaa1fbc71a606db0f
I20260621 07:50:44.898790 17716 raft_consensus.cc:2726] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 07:50:44.916795 17717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:44.916439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.195:53105)
0621 07:50:44.916688 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:44.916730 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":62,"threads_started":1}
I20260621 07:50:45.650441 17718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:45.650063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.193:36057)
0621 07:50:45.650299 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:45.650351 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":72,"threads_started":1}
I20260621 07:50:45.749550 17719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:45.749171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.194:38651)
0621 07:50:45.749411 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:45.749462 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":74,"threads_started":1}
I20260621 07:50:45.917670 17720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:45.917347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.195:58017)
0621 07:50:45.917547 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:45.917594 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":59,"threads_started":1}
I20260621 07:50:46.651700 17721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:46.651274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.193:36613)
0621 07:50:46.651568 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:46.651615 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":89,"threads_started":1}
I20260621 07:50:46.750578 17722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:46.750228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.194:47955)
0621 07:50:46.750455 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:46.750502 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":85,"threads_started":1}
I20260621 07:50:46.918702 17723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:46.918359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.195:53279)
0621 07:50:46.918583 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:46.918629 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":88,"threads_started":1}
I20260621 07:50:47.352710 17724 raft_consensus.cc:482] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:50:47.352852 17724 raft_consensus.cc:504] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8737bbdc600e4cfbaa1fbc71a606db0f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43481 } } peers { permanent_uuid: "48ffd15f6b16463ebd049ab624d11205" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 45517 } } peers { permanent_uuid: "854c12bbec8949f397347e2dc7604940" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46373 } }
I20260621 07:50:47.353245 17724 leader_election.cc:288] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8737bbdc600e4cfbaa1fbc71a606db0f (127.0.192.194:43481), 48ffd15f6b16463ebd049ab624d11205 (127.0.192.195:45517)
W20260621 07:50:47.354032 17322 leader_election.cc:341] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8737bbdc600e4cfbaa1fbc71a606db0f (127.0.192.194:43481): Not found: Tablet not found: 4913ceae6fd745a4b261a9317aedf95e
W20260621 07:50:47.354120 17321 leader_election.cc:341] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 48ffd15f6b16463ebd049ab624d11205 (127.0.192.195:45517): Not found: Tablet not found: 4913ceae6fd745a4b261a9317aedf95e
I20260621 07:50:47.354176 17321 leader_election.cc:302] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 854c12bbec8949f397347e2dc7604940; no voters: 48ffd15f6b16463ebd049ab624d11205, 8737bbdc600e4cfbaa1fbc71a606db0f
I20260621 07:50:47.354351 17724 raft_consensus.cc:2726] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 07:50:47.652781 17725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:47.652344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.193:49487)
0621 07:50:47.652644 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:47.652690 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":91,"threads_started":1}
I20260621 07:50:47.751598 17726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:47.751272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.194:59871)
0621 07:50:47.751470 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:47.751520 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":61,"threads_started":1}
I20260621 07:50:47.919921 17727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:47.919522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.195:45399)
0621 07:50:47.919797 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:47.919848 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":90,"threads_started":1}
I20260621 07:50:48.654050 17728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:48.653587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.193:48313)
0621 07:50:48.653883 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:48.653951 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":92,"threads_started":1}
I20260621 07:50:48.752857 17729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:48.752333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.194:52213)
0621 07:50:48.752661 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:48.752759 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":68,"threads_started":1}
I20260621 07:50:48.921048 17730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:48.920611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.195:39337)
0621 07:50:48.920906 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:48.920961 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":74,"threads_started":1}
I20260621 07:50:49.370841 17731 raft_consensus.cc:482] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:50:49.370946 17731 raft_consensus.cc:504] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8737bbdc600e4cfbaa1fbc71a606db0f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43481 } } peers { permanent_uuid: "48ffd15f6b16463ebd049ab624d11205" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 45517 } } peers { permanent_uuid: "854c12bbec8949f397347e2dc7604940" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46373 } }
I20260621 07:50:49.371328 17731 leader_election.cc:288] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8737bbdc600e4cfbaa1fbc71a606db0f (127.0.192.194:43481), 48ffd15f6b16463ebd049ab624d11205 (127.0.192.195:45517)
W20260621 07:50:49.371968 17322 leader_election.cc:341] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8737bbdc600e4cfbaa1fbc71a606db0f (127.0.192.194:43481): Not found: Tablet not found: 4913ceae6fd745a4b261a9317aedf95e
W20260621 07:50:49.372046 17321 leader_election.cc:341] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 48ffd15f6b16463ebd049ab624d11205 (127.0.192.195:45517): Not found: Tablet not found: 4913ceae6fd745a4b261a9317aedf95e
I20260621 07:50:49.372145 17321 leader_election.cc:302] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 854c12bbec8949f397347e2dc7604940; no voters: 48ffd15f6b16463ebd049ab624d11205, 8737bbdc600e4cfbaa1fbc71a606db0f
I20260621 07:50:49.372256 17731 raft_consensus.cc:2726] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 07:50:49.655300 17732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:49.654960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.193:57979)
0621 07:50:49.655172 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:49.655217 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":74,"threads_started":1}
I20260621 07:50:49.753886 17733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:49.753528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.194:37857)
0621 07:50:49.753745 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:49.753795 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":73,"threads_started":1}
I20260621 07:50:49.921983 17734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:49.921667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.195:39373)
0621 07:50:49.921866 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:49.921912 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":56,"threads_started":1}
I20260621 07:50:50.656322 17735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:50.655971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.193:36997)
0621 07:50:50.656193 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:50.656238 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":89,"threads_started":1}
I20260621 07:50:50.754966 17736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:50.754604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.194:55701)
0621 07:50:50.754838 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:50.754885 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":79,"threads_started":1}
I20260621 07:50:50.922971 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:50.922596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.195:37153)
0621 07:50:50.922829 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:50.922883 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":75,"threads_started":1}
I20260621 07:50:51.657423 17738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:51.657032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.193:36117)
0621 07:50:51.657273 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:51.657329 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":68,"threads_started":1}
I20260621 07:50:51.755995 17739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:51.755651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.194:52225)
0621 07:50:51.755869 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:51.755915 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":68,"threads_started":1}
I20260621 07:50:51.924189 17740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:51.923728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.195:38631)
0621 07:50:51.924052 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:51.924102 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":76,"threads_started":1}
I20260621 07:50:52.658432 17741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:52.658072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.193:44357)
0621 07:50:52.658292 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:52.658338 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":78,"threads_started":1}
I20260621 07:50:52.757099 17742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:52.756753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.194:43511)
0621 07:50:52.756969 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:52.757017 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":67,"threads_started":1}
I20260621 07:50:52.896772 17743 raft_consensus.cc:482] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:50:52.896904 17743 raft_consensus.cc:504] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8737bbdc600e4cfbaa1fbc71a606db0f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43481 } } peers { permanent_uuid: "48ffd15f6b16463ebd049ab624d11205" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 45517 } } peers { permanent_uuid: "854c12bbec8949f397347e2dc7604940" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46373 } }
I20260621 07:50:52.897415 17743 leader_election.cc:288] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8737bbdc600e4cfbaa1fbc71a606db0f (127.0.192.194:43481), 48ffd15f6b16463ebd049ab624d11205 (127.0.192.195:45517)
W20260621 07:50:52.898074 17322 leader_election.cc:341] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8737bbdc600e4cfbaa1fbc71a606db0f (127.0.192.194:43481): Not found: Tablet not found: 4913ceae6fd745a4b261a9317aedf95e
W20260621 07:50:52.898183 17321 leader_election.cc:341] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 48ffd15f6b16463ebd049ab624d11205 (127.0.192.195:45517): Not found: Tablet not found: 4913ceae6fd745a4b261a9317aedf95e
I20260621 07:50:52.898299 17321 leader_election.cc:302] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 854c12bbec8949f397347e2dc7604940; no voters: 48ffd15f6b16463ebd049ab624d11205, 8737bbdc600e4cfbaa1fbc71a606db0f
I20260621 07:50:52.898480 17743 raft_consensus.cc:2726] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 07:50:52.925146 17744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:52.924852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.195:46323)
0621 07:50:52.925034 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:52.925079 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":52,"threads_started":1}
I20260621 07:50:53.659456 17745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:53.659092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.193:44027)
0621 07:50:53.659322 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:53.659368 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":81,"threads_started":1}
I20260621 07:50:53.758080 17746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:53.757737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.194:34089)
0621 07:50:53.757963 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:53.758008 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":79,"threads_started":1}
I20260621 07:50:53.926188 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:53.925751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.195:56091)
0621 07:50:53.926074 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:53.926120 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":73,"threads_started":1}
I20260621 07:50:54.536837 17748 raft_consensus.cc:482] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:50:54.536969 17748 raft_consensus.cc:504] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8737bbdc600e4cfbaa1fbc71a606db0f" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 43481 } } peers { permanent_uuid: "48ffd15f6b16463ebd049ab624d11205" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 45517 } } peers { permanent_uuid: "854c12bbec8949f397347e2dc7604940" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46373 } }
I20260621 07:50:54.537321 17748 leader_election.cc:288] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8737bbdc600e4cfbaa1fbc71a606db0f (127.0.192.194:43481), 48ffd15f6b16463ebd049ab624d11205 (127.0.192.195:45517)
W20260621 07:50:54.537969 17322 leader_election.cc:341] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8737bbdc600e4cfbaa1fbc71a606db0f (127.0.192.194:43481): Not found: Tablet not found: 4913ceae6fd745a4b261a9317aedf95e
W20260621 07:50:54.538057 17321 leader_election.cc:341] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 48ffd15f6b16463ebd049ab624d11205 (127.0.192.195:45517): Not found: Tablet not found: 4913ceae6fd745a4b261a9317aedf95e
I20260621 07:50:54.538161 17321 leader_election.cc:302] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 854c12bbec8949f397347e2dc7604940; no voters: 48ffd15f6b16463ebd049ab624d11205, 8737bbdc600e4cfbaa1fbc71a606db0f
I20260621 07:50:54.538327 17748 raft_consensus.cc:2726] T 4913ceae6fd745a4b261a9317aedf95e P 854c12bbec8949f397347e2dc7604940 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 07:50:54.660655 17749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:54.660237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.193:47495)
0621 07:50:54.660451 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:54.660497 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":59,"threads_started":1}
I20260621 07:50:54.759291 17750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:54.758933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.194:53211)
0621 07:50:54.759140 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:54.759204 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":78,"threads_started":1}
I20260621 07:50:54.927297 17751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:54.926980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.195:45509)
0621 07:50:54.927185 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:54.927229 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":71,"threads_started":1}
I20260621 07:50:55.661692 17752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:55.661318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.193:33747)
0621 07:50:55.661544 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:55.661589 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":70,"threads_started":1}
I20260621 07:50:55.760349 17753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:55.759950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.194:47647)
0621 07:50:55.760170 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:55.760259 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":68,"threads_started":1}
I20260621 07:50:55.928386 17754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:55.927988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.195:53299)
0621 07:50:55.928265 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:55.928312 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":69,"threads_started":1}
I20260621 07:50:56.662846 17755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:56.662492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.193:59451)
0621 07:50:56.662715 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:56.662762 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":83,"threads_started":1}
I20260621 07:50:56.761492 17756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:56.761135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45149 (local address 127.0.192.194:58383)
0621 07:50:56.761350 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:56.761409 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.254:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":67,"threads_started":1}
I20260621 07:50:56.917623 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 17264
I20260621 07:50:56.924197 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 17436
I20260621 07:50:56.928969 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 17571
2026-06-21T07:50:56Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58896 ms)
[----------] 30 tests from SecurityITest (228402 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:50:56 dist-test-slave-lzx3 krb5kdc[17759](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:50:56 dist-test-slave-lzx3 krb5kdc[17759](info): set up 2 sockets
Jun 21 07:50:56 dist-test-slave-lzx3 krb5kdc[17759](info): commencing operation
krb5kdc: starting...
W20260621 07:50:57.868018 771 mini_kdc.cc:121] Time spent starting KDC: real 0.918s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/krb5kdc/test-user.keytab.
Jun 21 07:50:57 dist-test-slave-lzx3 krb5kdc[17759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:50:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:50:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.192.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
I20260621 07:50:57.969135 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:39081
--webserver_interface=127.0.192.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:45309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:39081
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:50:58.066314 17772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:50:58.066538 17772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:50:58.066593 17772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:50:58.070447 17772 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:50:58.070510 17772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:50:58.070542 17772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:50:58.070585 17772 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:50:58.070614 17772 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:50:58.075028 17772 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:45309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:39081
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:39081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:50:58.075846 17772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:50:58.076738 17772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:50:58.084137 17777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:50:58.084121 17778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:50:58.084255 17772 server_base.cc:1034] running on GCE node
W20260621 07:50:58.084121 17780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:50:58.084892 17772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:50:58.085834 17772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:50:58.087033 17772 hybrid_clock.cc:648] HybridClock initialized: now 1782028258087018 us; error 28 us; skew 500 ppm
Jun 21 07:50:58 dist-test-slave-lzx3 krb5kdc[17759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:50:58.089565 17772 init.cc:377] Logged in from keytab as kudu/127.0.192.254@KRBTEST.COM (short username kudu)
I20260621 07:50:58.090308 17772 webserver.cc:460] Webserver started at http://127.0.192.254:37895/ using document root <none> and password file <none>
I20260621 07:50:58.090806 17772 fs_manager.cc:359] Metadata directory not provided
I20260621 07:50:58.090885 17772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:50:58.091077 17772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:50:58.092840 17772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "79e920e4a3fd4a11814bc75cfdfd7c2e"
format_stamp: "Formatted at 2026-06-21 07:50:58 on dist-test-slave-lzx3"
server_key: "d2a4e1c3cab9904a65da52f5a0b4fbe9"
server_key_iv: "b09a467842cf02834b4150819b2acea1"
server_key_version: "encryptionkey@0"
I20260621 07:50:58.093327 17772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "79e920e4a3fd4a11814bc75cfdfd7c2e"
format_stamp: "Formatted at 2026-06-21 07:50:58 on dist-test-slave-lzx3"
server_key: "d2a4e1c3cab9904a65da52f5a0b4fbe9"
server_key_iv: "b09a467842cf02834b4150819b2acea1"
server_key_version: "encryptionkey@0"
I20260621 07:50:58.096339 17772 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260621 07:50:58.098315 17787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:50:58.099313 17772 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260621 07:50:58.099442 17772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "79e920e4a3fd4a11814bc75cfdfd7c2e"
format_stamp: "Formatted at 2026-06-21 07:50:58 on dist-test-slave-lzx3"
server_key: "d2a4e1c3cab9904a65da52f5a0b4fbe9"
server_key_iv: "b09a467842cf02834b4150819b2acea1"
server_key_version: "encryptionkey@0"
I20260621 07:50:58.099543 17772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:50:58.113245 17772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:50:58.115451 17772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:50:58.115630 17772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:50:58.121719 17772 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:39081
I20260621 07:50:58.121806 17839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:39081 every 8 connection(s)
I20260621 07:50:58.122463 17772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:50:58.123754 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 17772
I20260621 07:50:58.123878 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:50:58.124153 771 external_mini_cluster.cc:1428] Setting key f88ecbe9e093ba604ff078df8a9ed1c3
I20260621 07:50:58.125257 17840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:50:58.130905 17840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79e920e4a3fd4a11814bc75cfdfd7c2e: Bootstrap starting.
Jun 21 07:50:58 dist-test-slave-lzx3 krb5kdc[17759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:50:58.132892 17840 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79e920e4a3fd4a11814bc75cfdfd7c2e: Neither blocks nor log segments found. Creating new log.
I20260621 07:50:58.133592 17840 log.cc:824] T 00000000000000000000000000000000 P 79e920e4a3fd4a11814bc75cfdfd7c2e: Log is configured to *not* fsync() on all Append() calls
I20260621 07:50:58.135092 17840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79e920e4a3fd4a11814bc75cfdfd7c2e: No bootstrap required, opened a new log
I20260621 07:50:58.136109 17843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:58.125376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38936 (local address 127.0.192.254:39081)
0621 07:50:58.125700 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0621 07:50:58.125716 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 07:50:58.125760 (+ 44us) server_negotiation.cc:407] Connection header received
0621 07:50:58.126255 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:50:58.126268 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:50:58.126518 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:50:58.126737 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:50:58.127507 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:58.128342 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:58.129432 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:58.129854 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:58.129966 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:58.132298 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:50:58.132302 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:50:58.132311 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:50:58.132332 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:50:58.133842 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:50:58.134228 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:50:58.134233 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:50:58.134247 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:50:58.134337 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:50:58.134590 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:50:58.134594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:50:58.134595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:50:58.134928 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:50:58.135105 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0621 07:50:58.135377 (+ 272us) server_negotiation.cc:299] Negotiation successful
0621 07:50:58.135588 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":115,"threads_started":1}
I20260621 07:50:58.138782 17840 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79e920e4a3fd4a11814bc75cfdfd7c2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79e920e4a3fd4a11814bc75cfdfd7c2e" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 39081 } }
I20260621 07:50:58.138998 17840 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79e920e4a3fd4a11814bc75cfdfd7c2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:50:58.139072 17840 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79e920e4a3fd4a11814bc75cfdfd7c2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79e920e4a3fd4a11814bc75cfdfd7c2e, State: Initialized, Role: FOLLOWER
I20260621 07:50:58.139531 17840 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79e920e4a3fd4a11814bc75cfdfd7c2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79e920e4a3fd4a11814bc75cfdfd7c2e" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 39081 } }
I20260621 07:50:58.139673 17840 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79e920e4a3fd4a11814bc75cfdfd7c2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:50:58.139765 17840 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79e920e4a3fd4a11814bc75cfdfd7c2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:50:58.139856 17840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79e920e4a3fd4a11814bc75cfdfd7c2e [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:50:58.140717 17840 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79e920e4a3fd4a11814bc75cfdfd7c2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79e920e4a3fd4a11814bc75cfdfd7c2e" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 39081 } }
I20260621 07:50:58.141022 17840 leader_election.cc:302] T 00000000000000000000000000000000 P 79e920e4a3fd4a11814bc75cfdfd7c2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79e920e4a3fd4a11814bc75cfdfd7c2e; no voters:
I20260621 07:50:58.141283 17840 leader_election.cc:288] T 00000000000000000000000000000000 P 79e920e4a3fd4a11814bc75cfdfd7c2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:50:58.141433 17845 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79e920e4a3fd4a11814bc75cfdfd7c2e [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:50:58.141629 17845 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79e920e4a3fd4a11814bc75cfdfd7c2e [term 1 LEADER]: Becoming Leader. State: Replica: 79e920e4a3fd4a11814bc75cfdfd7c2e, State: Running, Role: LEADER
I20260621 07:50:58.141954 17845 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79e920e4a3fd4a11814bc75cfdfd7c2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79e920e4a3fd4a11814bc75cfdfd7c2e" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 39081 } }
I20260621 07:50:58.142429 17840 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79e920e4a3fd4a11814bc75cfdfd7c2e [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:50:58.143465 17847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79e920e4a3fd4a11814bc75cfdfd7c2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79e920e4a3fd4a11814bc75cfdfd7c2e. Latest consensus state: current_term: 1 leader_uuid: "79e920e4a3fd4a11814bc75cfdfd7c2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79e920e4a3fd4a11814bc75cfdfd7c2e" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 39081 } } }
I20260621 07:50:58.143575 17847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79e920e4a3fd4a11814bc75cfdfd7c2e [sys.catalog]: This master's current role is: LEADER
I20260621 07:50:58.143900 17846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79e920e4a3fd4a11814bc75cfdfd7c2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79e920e4a3fd4a11814bc75cfdfd7c2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79e920e4a3fd4a11814bc75cfdfd7c2e" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 39081 } } }
I20260621 07:50:58.144050 17846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79e920e4a3fd4a11814bc75cfdfd7c2e [sys.catalog]: This master's current role is: LEADER
W20260621 07:50:58.146538 17861 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 79e920e4a3fd4a11814bc75cfdfd7c2e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:50:58.146595 17861 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:50:58.146664 17854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:50:58.147264 17854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:50:58.151942 17854 catalog_manager.cc:1347] Generated new cluster ID: 9b1583ea10d143e79fc7ddf35c766981
I20260621 07:50:58.151991 17854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:50:58.172945 17854 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:50:58.173939 17854 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:50:58.183846 17854 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79e920e4a3fd4a11814bc75cfdfd7c2e: Generated new TSK 0
I20260621 07:50:58.184440 17854 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.192.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab.
Entry for principal kudu/127.0.192.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab.
I20260621 07:50:58.211511 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.193:0
--local_ip_for_outbound_sockets=127.0.192.193
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:39081
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:45309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:50:58.309816 17866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:50:58.310039 17866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:50:58.310130 17866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:50:58.313700 17866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:50:58.313746 17866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:50:58.313855 17866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.193
I20260621 07:50:58.318929 17866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:45309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:39081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.193
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:50:58.319782 17866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:50:58.320700 17866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:50:58.328821 17872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:50:58.328832 17871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:50:58.328832 17874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:50:58.328924 17866 server_base.cc:1034] running on GCE node
I20260621 07:50:58.329511 17866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:50:58.330056 17866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:50:58.331223 17866 hybrid_clock.cc:648] HybridClock initialized: now 1782028258331197 us; error 41 us; skew 500 ppm
Jun 21 07:50:58 dist-test-slave-lzx3 krb5kdc[17759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:50:58.333703 17866 init.cc:377] Logged in from keytab as kudu/127.0.192.193@KRBTEST.COM (short username kudu)
I20260621 07:50:58.334432 17866 webserver.cc:460] Webserver started at http://127.0.192.193:41475/ using document root <none> and password file <none>
I20260621 07:50:58.334909 17866 fs_manager.cc:359] Metadata directory not provided
I20260621 07:50:58.335005 17866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:50:58.335198 17866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:50:58.337074 17866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-0/data/instance:
uuid: "cd0294f749504ea883195192c19487a9"
format_stamp: "Formatted at 2026-06-21 07:50:58 on dist-test-slave-lzx3"
server_key: "73e421ed0da7067940d01105a90ae7ac"
server_key_iv: "a71edf575d357f7f385396a5cbb6b985"
server_key_version: "encryptionkey@0"
I20260621 07:50:58.337618 17866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-0/wal/instance:
uuid: "cd0294f749504ea883195192c19487a9"
format_stamp: "Formatted at 2026-06-21 07:50:58 on dist-test-slave-lzx3"
server_key: "73e421ed0da7067940d01105a90ae7ac"
server_key_iv: "a71edf575d357f7f385396a5cbb6b985"
server_key_version: "encryptionkey@0"
I20260621 07:50:58.341092 17866 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:50:58.343621 17881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:50:58.344961 17866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:50:58.345091 17866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-0/wal
uuid: "cd0294f749504ea883195192c19487a9"
format_stamp: "Formatted at 2026-06-21 07:50:58 on dist-test-slave-lzx3"
server_key: "73e421ed0da7067940d01105a90ae7ac"
server_key_iv: "a71edf575d357f7f385396a5cbb6b985"
server_key_version: "encryptionkey@0"
I20260621 07:50:58.345196 17866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:50:58.377223 17866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:50:58.379674 17866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:50:58.379861 17866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:50:58.380383 17866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:50:58.381181 17866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:50:58.381244 17866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:50:58.381325 17866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:50:58.381371 17866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:50:58.389729 17866 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.193:39139
I20260621 07:50:58.389777 17994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.193:39139 every 8 connection(s)
I20260621 07:50:58.390501 17866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
I20260621 07:50:58.396683 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 17866
I20260621 07:50:58.396806 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-0/wal/instance
I20260621 07:50:58.397089 771 external_mini_cluster.cc:1428] Setting key 59ce0bc7278d2c536afa3b2f8320cd86
Jun 21 07:50:58 dist-test-slave-lzx3 krb5kdc[17759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:50:58.402638 17843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:58.391989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:57517 (local address 127.0.192.254:39081)
0621 07:50:58.392176 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0621 07:50:58.392181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:50:58.392890 (+ 709us) server_negotiation.cc:407] Connection header received
0621 07:50:58.393945 (+ 1055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:50:58.393947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:50:58.393988 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:50:58.394053 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:50:58.394981 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:58.395483 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:58.396474 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:58.396832 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:58.396934 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:58.399081 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:50:58.399084 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:50:58.399086 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:50:58.399107 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:50:58.400779 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:50:58.401315 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:50:58.401318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:50:58.401319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:50:58.401385 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:50:58.401732 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:50:58.401734 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:50:58.401735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:50:58.401876 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:50:58.401944 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0621 07:50:58.402455 (+ 511us) server_negotiation.cc:299] Negotiation successful
0621 07:50:58.402541 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260621 07:50:58.403386 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:58.392340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:39081 (local address 127.0.192.193:57517)
0621 07:50:58.392748 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:58.392793 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 07:50:58.393780 (+ 987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:50:58.394188 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:50:58.394194 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:50:58.394481 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:50:58.394869 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:58.394880 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:58.395580 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:58.395583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:58.396363 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:58.396370 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:58.396961 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:58.396964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:58.397082 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:58.397585 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:50:58.397599 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:50:58.398939 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:50:58.400888 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:50:58.400892 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:50:58.400901 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:50:58.401217 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:50:58.401467 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:50:58.401472 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:50:58.401477 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:50:58.401650 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:50:58.401980 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:50:58.401988 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:50:58.402240 (+ 252us) client_negotiation.cc:769] Sending connection context
0621 07:50:58.402429 (+ 189us) client_negotiation.cc:240] Negotiation successful
0621 07:50:58.402716 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":99,"threads_started":1}
I20260621 07:50:58.404424 17995 heartbeater.cc:347] Connected to a master server at 127.0.192.254:39081
I20260621 07:50:58.404687 17995 heartbeater.cc:464] Registering TS with master...
I20260621 07:50:58.405164 17995 heartbeater.cc:511] Master 127.0.192.254:39081 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.192.194@KRBTEST.COM; defaulting to no policy
I20260621 07:50:58.406507 17804 ts_manager.cc:194] Registered new tserver with Master: cd0294f749504ea883195192c19487a9 (127.0.192.193:39139)
I20260621 07:50:58.407577 17804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.193@KRBTEST.COM'} at 127.0.192.193:57517
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab.
Entry for principal kudu/127.0.192.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab.
I20260621 07:50:58.421317 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.194:0
--local_ip_for_outbound_sockets=127.0.192.194
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:39081
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:45309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:50:58.517836 18000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:50:58.518023 18000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:50:58.518055 18000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:50:58.521946 18000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:50:58.521994 18000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:50:58.522097 18000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.194
I20260621 07:50:58.526736 18000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:45309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:39081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.194
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:50:58.527460 18000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:50:58.528230 18000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:50:58.536237 18006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:50:58.536259 18008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:50:58.536243 18005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:50:58.536270 18000 server_base.cc:1034] running on GCE node
I20260621 07:50:58.536906 18000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:50:58.537448 18000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:50:58.538630 18000 hybrid_clock.cc:648] HybridClock initialized: now 1782028258538622 us; error 26 us; skew 500 ppm
Jun 21 07:50:58 dist-test-slave-lzx3 krb5kdc[17759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:50:58.541167 18000 init.cc:377] Logged in from keytab as kudu/127.0.192.194@KRBTEST.COM (short username kudu)
I20260621 07:50:58.541893 18000 webserver.cc:460] Webserver started at http://127.0.192.194:44655/ using document root <none> and password file <none>
I20260621 07:50:58.542452 18000 fs_manager.cc:359] Metadata directory not provided
I20260621 07:50:58.542538 18000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:50:58.542750 18000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:50:58.544654 18000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-1/data/instance:
uuid: "f724e6df87394f438fae8a4682986b5e"
format_stamp: "Formatted at 2026-06-21 07:50:58 on dist-test-slave-lzx3"
server_key: "e9bf6ba89e7fe033ca945372be3e87cb"
server_key_iv: "dea0df07bb939972ed6ab2e57f2ae771"
server_key_version: "encryptionkey@0"
I20260621 07:50:58.545230 18000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-1/wal/instance:
uuid: "f724e6df87394f438fae8a4682986b5e"
format_stamp: "Formatted at 2026-06-21 07:50:58 on dist-test-slave-lzx3"
server_key: "e9bf6ba89e7fe033ca945372be3e87cb"
server_key_iv: "dea0df07bb939972ed6ab2e57f2ae771"
server_key_version: "encryptionkey@0"
I20260621 07:50:58.548448 18000 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260621 07:50:58.550690 18015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:50:58.551803 18000 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:50:58.551914 18000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-1/wal
uuid: "f724e6df87394f438fae8a4682986b5e"
format_stamp: "Formatted at 2026-06-21 07:50:58 on dist-test-slave-lzx3"
server_key: "e9bf6ba89e7fe033ca945372be3e87cb"
server_key_iv: "dea0df07bb939972ed6ab2e57f2ae771"
server_key_version: "encryptionkey@0"
I20260621 07:50:58.552002 18000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:50:58.578298 18000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:50:58.581272 18000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:50:58.581444 18000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:50:58.581955 18000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:50:58.582660 18000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:50:58.582731 18000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:50:58.582798 18000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:50:58.582830 18000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:50:58.590983 18000 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.194:45103
I20260621 07:50:58.591009 18128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.194:45103 every 8 connection(s)
I20260621 07:50:58.591739 18000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
I20260621 07:50:58.595877 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 18000
I20260621 07:50:58.595995 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-1/wal/instance
I20260621 07:50:58.596269 771 external_mini_cluster.cc:1428] Setting key c3954182b455ca19e0be79589414ade1
Jun 21 07:50:58 dist-test-slave-lzx3 krb5kdc[17759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:50:58.605026 17843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:58.593256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:38479 (local address 127.0.192.254:39081)
0621 07:50:58.593385 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0621 07:50:58.593389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:50:58.594154 (+ 765us) server_negotiation.cc:407] Connection header received
0621 07:50:58.595192 (+ 1038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:50:58.595194 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:50:58.595237 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:50:58.595303 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:50:58.596477 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:58.597196 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:58.598613 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:58.598824 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:58.598900 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:58.601638 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:50:58.601640 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:50:58.601642 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:50:58.601662 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:50:58.602965 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:50:58.603536 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:50:58.603538 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:50:58.603539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:50:58.603604 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:50:58.603985 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:50:58.603987 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:50:58.603988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:50:58.604147 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:50:58.604218 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0621 07:50:58.604833 (+ 615us) server_negotiation.cc:299] Negotiation successful
0621 07:50:58.604915 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
WARNING: no policy specified for kudu/127.0.192.195@KRBTEST.COM; defaulting to no policy
I20260621 07:50:58.605947 18131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:58.593611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:39081 (local address 127.0.192.194:38479)
0621 07:50:58.593982 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:58.594041 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0621 07:50:58.595028 (+ 987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:50:58.595411 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:50:58.595416 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:50:58.595750 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:50:58.596327 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:58.596347 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:58.597315 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:58.597317 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:58.598504 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:58.598515 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:58.598955 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:58.598961 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:58.599209 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:58.599807 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:50:58.599822 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:50:58.601478 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:50:58.603069 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:50:58.603076 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:50:58.603087 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:50:58.603439 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:50:58.603687 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:50:58.603690 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:50:58.603692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:50:58.603885 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:50:58.604249 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:50:58.604258 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:50:58.604532 (+ 274us) client_negotiation.cc:769] Sending connection context
0621 07:50:58.604804 (+ 272us) client_negotiation.cc:240] Negotiation successful
0621 07:50:58.605104 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":103,"threads_started":1}
I20260621 07:50:58.606907 18129 heartbeater.cc:347] Connected to a master server at 127.0.192.254:39081
I20260621 07:50:58.607077 18129 heartbeater.cc:464] Registering TS with master...
I20260621 07:50:58.607481 18129 heartbeater.cc:511] Master 127.0.192.254:39081 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.195@KRBTEST.COM" created.
I20260621 07:50:58.608371 17804 ts_manager.cc:194] Registered new tserver with Master: f724e6df87394f438fae8a4682986b5e (127.0.192.194:45103)
I20260621 07:50:58.609093 17804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.194@KRBTEST.COM'} at 127.0.192.194:38479
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab.
Entry for principal kudu/127.0.192.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab.
I20260621 07:50:58.620893 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.195:0
--local_ip_for_outbound_sockets=127.0.192.195
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:39081
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:45309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:50:58.717620 18134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:50:58.717804 18134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:50:58.717864 18134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:50:58.721549 18134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:50:58.721592 18134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:50:58.721691 18134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.195
I20260621 07:50:58.726384 18134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:45309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:39081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.195
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:50:58.727152 18134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:50:58.727977 18134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:50:58.736090 18139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:50:58.736192 18134 server_base.cc:1034] running on GCE node
W20260621 07:50:58.736090 18140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:50:58.736136 18142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:50:58.736809 18134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:50:58.737367 18134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:50:58.738552 18134 hybrid_clock.cc:648] HybridClock initialized: now 1782028258738537 us; error 27 us; skew 500 ppm
Jun 21 07:50:58 dist-test-slave-lzx3 krb5kdc[17759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:50:58.741459 18134 init.cc:377] Logged in from keytab as kudu/127.0.192.195@KRBTEST.COM (short username kudu)
I20260621 07:50:58.742242 18134 webserver.cc:460] Webserver started at http://127.0.192.195:37319/ using document root <none> and password file <none>
I20260621 07:50:58.742852 18134 fs_manager.cc:359] Metadata directory not provided
I20260621 07:50:58.743000 18134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:50:58.743191 18134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:50:58.745229 18134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-2/data/instance:
uuid: "1e0a827ef2884d8a81ad2bf9ca9f6cea"
format_stamp: "Formatted at 2026-06-21 07:50:58 on dist-test-slave-lzx3"
server_key: "6f2153191024527e457aaf03e63e66e6"
server_key_iv: "be6e9086255e603659b7529b353f7d91"
server_key_version: "encryptionkey@0"
I20260621 07:50:58.745728 18134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-2/wal/instance:
uuid: "1e0a827ef2884d8a81ad2bf9ca9f6cea"
format_stamp: "Formatted at 2026-06-21 07:50:58 on dist-test-slave-lzx3"
server_key: "6f2153191024527e457aaf03e63e66e6"
server_key_iv: "be6e9086255e603659b7529b353f7d91"
server_key_version: "encryptionkey@0"
I20260621 07:50:58.749001 18134 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:50:58.751142 18149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:50:58.752158 18134 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:50:58.752277 18134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-2/wal
uuid: "1e0a827ef2884d8a81ad2bf9ca9f6cea"
format_stamp: "Formatted at 2026-06-21 07:50:58 on dist-test-slave-lzx3"
server_key: "6f2153191024527e457aaf03e63e66e6"
server_key_iv: "be6e9086255e603659b7529b353f7d91"
server_key_version: "encryptionkey@0"
I20260621 07:50:58.752368 18134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:50:58.770212 18134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:50:58.772446 18134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:50:58.772662 18134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:50:58.773136 18134 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:50:58.773795 18134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:50:58.773859 18134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:50:58.773917 18134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:50:58.773957 18134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:50:58.781935 18134 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.195:44153
I20260621 07:50:58.781981 18262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.195:44153 every 8 connection(s)
I20260621 07:50:58.782629 18134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
I20260621 07:50:58.785521 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 18134
I20260621 07:50:58.785638 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-2/wal/instance
I20260621 07:50:58.785884 771 external_mini_cluster.cc:1428] Setting key 450b79333a0e78546f508529cc144ccc
Jun 21 07:50:58 dist-test-slave-lzx3 krb5kdc[17759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:50:58.794011 17843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:58.784058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:60941 (local address 127.0.192.254:39081)
0621 07:50:58.784189 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0621 07:50:58.784194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:50:58.784923 (+ 729us) server_negotiation.cc:407] Connection header received
0621 07:50:58.786080 (+ 1157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:50:58.786083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:50:58.786136 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:50:58.786207 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:50:58.787118 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:58.787649 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:58.788706 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:58.788937 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:58.789055 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:58.791099 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:50:58.791103 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:50:58.791105 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:50:58.791133 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:50:58.792162 (+ 1029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:50:58.792659 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:50:58.792662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:50:58.792664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:50:58.792737 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:50:58.793049 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:50:58.793051 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:50:58.793053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:50:58.793230 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:50:58.793331 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0621 07:50:58.793759 (+ 428us) server_negotiation.cc:299] Negotiation successful
0621 07:50:58.793861 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 07:50:58.794584 18265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:58.784410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:39081 (local address 127.0.192.195:60941)
0621 07:50:58.784801 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:58.784836 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 07:50:58.785881 (+ 1045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:50:58.786341 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:50:58.786347 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:50:58.786628 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:50:58.786994 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:58.787004 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:58.787757 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:58.787761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:58.788525 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:58.788532 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:58.789081 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:58.789084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:58.789245 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:58.789655 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:50:58.789669 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:50:58.790955 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:50:58.792264 (+ 1309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:50:58.792269 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:50:58.792279 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:50:58.792530 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:50:58.792831 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:50:58.792834 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:50:58.792836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:50:58.792953 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:50:58.793329 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:50:58.793335 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:50:58.793540 (+ 205us) client_negotiation.cc:769] Sending connection context
0621 07:50:58.793708 (+ 168us) client_negotiation.cc:240] Negotiation successful
0621 07:50:58.793955 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":148,"threads_started":1}
I20260621 07:50:58.795455 18263 heartbeater.cc:347] Connected to a master server at 127.0.192.254:39081
I20260621 07:50:58.795657 18263 heartbeater.cc:464] Registering TS with master...
I20260621 07:50:58.796036 18263 heartbeater.cc:511] Master 127.0.192.254:39081 requested a full tablet report, sending...
I20260621 07:50:58.796885 17804 ts_manager.cc:194] Registered new tserver with Master: 1e0a827ef2884d8a81ad2bf9ca9f6cea (127.0.192.195:44153)
I20260621 07:50:58.797375 17804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.195@KRBTEST.COM'} at 127.0.192.195:60941
I20260621 07:50:58.799207 771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:50:58.807529 17843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:58.800982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38950 (local address 127.0.192.254:39081)
0621 07:50:58.801102 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0621 07:50:58.801106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:50:58.801261 (+ 155us) server_negotiation.cc:407] Connection header received
0621 07:50:58.801388 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:50:58.801390 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:50:58.801431 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:50:58.801506 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:50:58.802101 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:58.802564 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:58.803416 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:58.803623 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:58.803725 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:58.804921 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:50:58.804925 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:50:58.804927 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:50:58.804949 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:50:58.806024 (+ 1075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:50:58.806412 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:50:58.806416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:50:58.806421 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:50:58.806497 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:50:58.806753 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:50:58.806758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:50:58.806759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:50:58.806947 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:50:58.807078 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0621 07:50:58.807298 (+ 220us) server_negotiation.cc:299] Negotiation successful
0621 07:50:58.807391 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260621 07:50:58.809815 17804 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:50:58.812373 17804 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:50:58.824020 18274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:58.819106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:45103 (local address 127.0.0.1:39894)
0621 07:50:58.819369 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:58.819408 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0621 07:50:58.819511 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:50:58.820252 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:50:58.820257 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:50:58.820275 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:50:58.820371 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:58.820380 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:58.821589 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:58.821591 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:58.822659 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:58.822667 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:58.823591 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:58.823593 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:58.823683 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:58.823713 (+ 30us) client_negotiation.cc:769] Sending connection context
0621 07:50:58.823804 (+ 91us) client_negotiation.cc:240] Negotiation successful
0621 07:50:58.823908 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":78,"threads_started":1}
I20260621 07:50:58.824185 18275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:58.819234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39894 (local address 127.0.192.194:45103)
0621 07:50:58.819568 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0621 07:50:58.819578 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:50:58.819597 (+ 19us) server_negotiation.cc:407] Connection header received
0621 07:50:58.819656 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:50:58.819659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:50:58.819892 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:50:58.820052 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:50:58.820498 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:58.821471 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:58.822761 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:58.823486 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:58.823590 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:58.823712 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0621 07:50:58.823841 (+ 129us) server_negotiation.cc:299] Negotiation successful
0621 07:50:58.823937 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":155,"threads_started":1}
I20260621 07:50:58.825016 18277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:58.819874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52932 (local address 127.0.192.195:44153)
0621 07:50:58.820874 (+ 1000us) server_negotiation.cc:206] Beginning negotiation
0621 07:50:58.820885 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 07:50:58.820907 (+ 22us) server_negotiation.cc:407] Connection header received
0621 07:50:58.820949 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:50:58.820952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:50:58.821058 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:50:58.821192 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:50:58.821408 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:58.822355 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:58.823582 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:58.824374 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:58.824678 (+ 304us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:58.824785 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0621 07:50:58.824829 (+ 44us) server_negotiation.cc:299] Negotiation successful
0621 07:50:58.824894 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":928,"thread_start_us":103,"threads_started":1}
I20260621 07:50:58.825066 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:58.819725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.195:44153 (local address 127.0.0.1:52932)
0621 07:50:58.820070 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:58.820083 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:50:58.820182 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:50:58.821176 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:50:58.821179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:50:58.821194 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:50:58.821311 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:58.821317 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:58.822466 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:58.822469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:58.823458 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:58.823467 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:58.824494 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:58.824497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:58.824700 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:58.824741 (+ 41us) client_negotiation.cc:769] Sending connection context
0621 07:50:58.824804 (+ 63us) client_negotiation.cc:240] Negotiation successful
0621 07:50:58.824933 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":106,"thread_start_us":104,"threads_started":1}
I20260621 07:50:58.825186 18278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:58.820155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:39139 (local address 127.0.0.1:45574)
0621 07:50:58.820351 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:58.820362 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 07:50:58.820431 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:50:58.821243 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:50:58.821246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:50:58.821260 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:50:58.821363 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:58.821368 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:58.822360 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:58.822364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:58.823458 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:58.823467 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:58.824475 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:58.824479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:58.824616 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:58.824742 (+ 126us) client_negotiation.cc:769] Sending connection context
0621 07:50:58.824806 (+ 64us) client_negotiation.cc:240] Negotiation successful
0621 07:50:58.824933 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":180,"thread_start_us":77,"threads_started":1}
I20260621 07:50:58.825479 18279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:58.820523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45574 (local address 127.0.192.193:39139)
0621 07:50:58.820860 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0621 07:50:58.820869 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:50:58.820888 (+ 19us) server_negotiation.cc:407] Connection header received
0621 07:50:58.820957 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:50:58.820962 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:50:58.821114 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:50:58.821473 (+ 359us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:50:58.821619 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:58.822245 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:58.823580 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:58.824354 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:58.824475 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:58.824630 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0621 07:50:58.825303 (+ 673us) server_negotiation.cc:299] Negotiation successful
0621 07:50:58.825384 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":106,"threads_started":1}
I20260621 07:50:58.826761 18063 tablet_service.cc:1467] Processing CreateTablet for tablet 104b7473444e49b884e663bf131fbf01 (DEFAULT_TABLE table=test-table [id=9ec5b38166c04a55ac8966c0375c45a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:50:58.827493 18063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 104b7473444e49b884e663bf131fbf01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:50:58.831122 18197 tablet_service.cc:1467] Processing CreateTablet for tablet 104b7473444e49b884e663bf131fbf01 (DEFAULT_TABLE table=test-table [id=9ec5b38166c04a55ac8966c0375c45a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:50:58.831122 17929 tablet_service.cc:1467] Processing CreateTablet for tablet 104b7473444e49b884e663bf131fbf01 (DEFAULT_TABLE table=test-table [id=9ec5b38166c04a55ac8966c0375c45a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:50:58.831939 17929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 104b7473444e49b884e663bf131fbf01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:50:58.831944 18197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 104b7473444e49b884e663bf131fbf01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:50:58.833117 18280 tablet_bootstrap.cc:492] T 104b7473444e49b884e663bf131fbf01 P f724e6df87394f438fae8a4682986b5e: Bootstrap starting.
I20260621 07:50:58.835359 18280 tablet_bootstrap.cc:654] T 104b7473444e49b884e663bf131fbf01 P f724e6df87394f438fae8a4682986b5e: Neither blocks nor log segments found. Creating new log.
I20260621 07:50:58.835963 18280 log.cc:824] T 104b7473444e49b884e663bf131fbf01 P f724e6df87394f438fae8a4682986b5e: Log is configured to *not* fsync() on all Append() calls
I20260621 07:50:58.837556 18280 tablet_bootstrap.cc:492] T 104b7473444e49b884e663bf131fbf01 P f724e6df87394f438fae8a4682986b5e: No bootstrap required, opened a new log
I20260621 07:50:58.837749 18280 ts_tablet_manager.cc:1397] T 104b7473444e49b884e663bf131fbf01 P f724e6df87394f438fae8a4682986b5e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260621 07:50:58.838153 18283 tablet_bootstrap.cc:492] T 104b7473444e49b884e663bf131fbf01 P cd0294f749504ea883195192c19487a9: Bootstrap starting.
I20260621 07:50:58.838551 18282 tablet_bootstrap.cc:492] T 104b7473444e49b884e663bf131fbf01 P 1e0a827ef2884d8a81ad2bf9ca9f6cea: Bootstrap starting.
I20260621 07:50:58.840662 18282 tablet_bootstrap.cc:654] T 104b7473444e49b884e663bf131fbf01 P 1e0a827ef2884d8a81ad2bf9ca9f6cea: Neither blocks nor log segments found. Creating new log.
I20260621 07:50:58.841739 18282 log.cc:824] T 104b7473444e49b884e663bf131fbf01 P 1e0a827ef2884d8a81ad2bf9ca9f6cea: Log is configured to *not* fsync() on all Append() calls
I20260621 07:50:58.841790 18280 raft_consensus.cc:348] T 104b7473444e49b884e663bf131fbf01 P f724e6df87394f438fae8a4682986b5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f724e6df87394f438fae8a4682986b5e" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 45103 } } peers { permanent_uuid: "1e0a827ef2884d8a81ad2bf9ca9f6cea" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44153 } } peers { permanent_uuid: "cd0294f749504ea883195192c19487a9" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 39139 } }
I20260621 07:50:58.842051 18280 raft_consensus.cc:374] T 104b7473444e49b884e663bf131fbf01 P f724e6df87394f438fae8a4682986b5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:50:58.842114 18280 raft_consensus.cc:729] T 104b7473444e49b884e663bf131fbf01 P f724e6df87394f438fae8a4682986b5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f724e6df87394f438fae8a4682986b5e, State: Initialized, Role: FOLLOWER
I20260621 07:50:58.842527 18283 tablet_bootstrap.cc:654] T 104b7473444e49b884e663bf131fbf01 P cd0294f749504ea883195192c19487a9: Neither blocks nor log segments found. Creating new log.
I20260621 07:50:58.842526 18280 consensus_queue.cc:260] T 104b7473444e49b884e663bf131fbf01 P f724e6df87394f438fae8a4682986b5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f724e6df87394f438fae8a4682986b5e" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 45103 } } peers { permanent_uuid: "1e0a827ef2884d8a81ad2bf9ca9f6cea" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44153 } } peers { permanent_uuid: "cd0294f749504ea883195192c19487a9" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 39139 } }
I20260621 07:50:58.843264 18129 heartbeater.cc:503] Master 127.0.192.254:39081 was elected leader, sending a full tablet report...
I20260621 07:50:58.843392 18283 log.cc:824] T 104b7473444e49b884e663bf131fbf01 P cd0294f749504ea883195192c19487a9: Log is configured to *not* fsync() on all Append() calls
I20260621 07:50:58.843472 18282 tablet_bootstrap.cc:492] T 104b7473444e49b884e663bf131fbf01 P 1e0a827ef2884d8a81ad2bf9ca9f6cea: No bootstrap required, opened a new log
I20260621 07:50:58.843631 18282 ts_tablet_manager.cc:1397] T 104b7473444e49b884e663bf131fbf01 P 1e0a827ef2884d8a81ad2bf9ca9f6cea: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260621 07:50:58.844738 18280 ts_tablet_manager.cc:1428] T 104b7473444e49b884e663bf131fbf01 P f724e6df87394f438fae8a4682986b5e: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260621 07:50:58.845202 18283 tablet_bootstrap.cc:492] T 104b7473444e49b884e663bf131fbf01 P cd0294f749504ea883195192c19487a9: No bootstrap required, opened a new log
I20260621 07:50:58.845371 18283 ts_tablet_manager.cc:1397] T 104b7473444e49b884e663bf131fbf01 P cd0294f749504ea883195192c19487a9: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260621 07:50:58.846688 18282 raft_consensus.cc:348] T 104b7473444e49b884e663bf131fbf01 P 1e0a827ef2884d8a81ad2bf9ca9f6cea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f724e6df87394f438fae8a4682986b5e" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 45103 } } peers { permanent_uuid: "1e0a827ef2884d8a81ad2bf9ca9f6cea" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44153 } } peers { permanent_uuid: "cd0294f749504ea883195192c19487a9" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 39139 } }
I20260621 07:50:58.846949 18282 raft_consensus.cc:374] T 104b7473444e49b884e663bf131fbf01 P 1e0a827ef2884d8a81ad2bf9ca9f6cea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:50:58.847002 18282 raft_consensus.cc:729] T 104b7473444e49b884e663bf131fbf01 P 1e0a827ef2884d8a81ad2bf9ca9f6cea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e0a827ef2884d8a81ad2bf9ca9f6cea, State: Initialized, Role: FOLLOWER
I20260621 07:50:58.847416 18282 consensus_queue.cc:260] T 104b7473444e49b884e663bf131fbf01 P 1e0a827ef2884d8a81ad2bf9ca9f6cea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f724e6df87394f438fae8a4682986b5e" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 45103 } } peers { permanent_uuid: "1e0a827ef2884d8a81ad2bf9ca9f6cea" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44153 } } peers { permanent_uuid: "cd0294f749504ea883195192c19487a9" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 39139 } }
I20260621 07:50:58.848052 18263 heartbeater.cc:503] Master 127.0.192.254:39081 was elected leader, sending a full tablet report...
I20260621 07:50:58.848326 18282 ts_tablet_manager.cc:1428] T 104b7473444e49b884e663bf131fbf01 P 1e0a827ef2884d8a81ad2bf9ca9f6cea: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260621 07:50:58.849124 18283 raft_consensus.cc:348] T 104b7473444e49b884e663bf131fbf01 P cd0294f749504ea883195192c19487a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f724e6df87394f438fae8a4682986b5e" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 45103 } } peers { permanent_uuid: "1e0a827ef2884d8a81ad2bf9ca9f6cea" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44153 } } peers { permanent_uuid: "cd0294f749504ea883195192c19487a9" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 39139 } }
I20260621 07:50:58.849403 18283 raft_consensus.cc:374] T 104b7473444e49b884e663bf131fbf01 P cd0294f749504ea883195192c19487a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:50:58.849478 18283 raft_consensus.cc:729] T 104b7473444e49b884e663bf131fbf01 P cd0294f749504ea883195192c19487a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd0294f749504ea883195192c19487a9, State: Initialized, Role: FOLLOWER
I20260621 07:50:58.849869 18283 consensus_queue.cc:260] T 104b7473444e49b884e663bf131fbf01 P cd0294f749504ea883195192c19487a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f724e6df87394f438fae8a4682986b5e" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 45103 } } peers { permanent_uuid: "1e0a827ef2884d8a81ad2bf9ca9f6cea" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44153 } } peers { permanent_uuid: "cd0294f749504ea883195192c19487a9" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 39139 } }
I20260621 07:50:58.850472 17995 heartbeater.cc:503] Master 127.0.192.254:39081 was elected leader, sending a full tablet report...
I20260621 07:50:58.850729 18283 ts_tablet_manager.cc:1428] T 104b7473444e49b884e663bf131fbf01 P cd0294f749504ea883195192c19487a9: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260621 07:50:58.891537 17996 tablet.cc:2369] T 104b7473444e49b884e663bf131fbf01 P cd0294f749504ea883195192c19487a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:50:58.988101 18287 raft_consensus.cc:482] T 104b7473444e49b884e663bf131fbf01 P 1e0a827ef2884d8a81ad2bf9ca9f6cea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:50:58.988300 18287 raft_consensus.cc:504] T 104b7473444e49b884e663bf131fbf01 P 1e0a827ef2884d8a81ad2bf9ca9f6cea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f724e6df87394f438fae8a4682986b5e" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 45103 } } peers { permanent_uuid: "1e0a827ef2884d8a81ad2bf9ca9f6cea" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44153 } } peers { permanent_uuid: "cd0294f749504ea883195192c19487a9" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 39139 } }
I20260621 07:50:58.989459 18287 leader_election.cc:288] T 104b7473444e49b884e663bf131fbf01 P 1e0a827ef2884d8a81ad2bf9ca9f6cea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f724e6df87394f438fae8a4682986b5e (127.0.192.194:45103), cd0294f749504ea883195192c19487a9 (127.0.192.193:39139)
I20260621 07:50:58.994071 18279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:58.990177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:43045 (local address 127.0.192.193:39139)
0621 07:50:58.990328 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0621 07:50:58.990333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:50:58.990491 (+ 158us) server_negotiation.cc:407] Connection header received
0621 07:50:58.990608 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:50:58.990614 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:50:58.990663 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:50:58.990734 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:50:58.990967 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:58.991594 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:58.992888 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:58.993477 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:58.993596 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:58.993643 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0621 07:50:58.993884 (+ 241us) server_negotiation.cc:299] Negotiation successful
0621 07:50:58.993948 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 07:50:58.994071 18275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:58.989801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:39893 (local address 127.0.192.194:45103)
0621 07:50:58.989943 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0621 07:50:58.989950 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:50:58.989967 (+ 17us) server_negotiation.cc:407] Connection header received
0621 07:50:58.990147 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:50:58.990150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:50:58.990203 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:50:58.990295 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:50:58.990646 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:58.991411 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:58.992888 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:58.993491 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:58.993595 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:58.993643 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0621 07:50:58.993884 (+ 241us) server_negotiation.cc:299] Negotiation successful
0621 07:50:58.993948 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 07:50:58.994117 18265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:58.989724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:45103 (local address 127.0.192.195:39893)
0621 07:50:58.989897 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:58.989915 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:50:58.990036 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:50:58.990374 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:50:58.990377 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:50:58.990396 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:50:58.990535 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:58.990545 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:58.991510 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:58.991511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:58.992772 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:58.992782 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:58.993593 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:58.993597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:58.993740 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:58.993776 (+ 36us) client_negotiation.cc:769] Sending connection context
0621 07:50:58.993852 (+ 76us) client_negotiation.cc:240] Negotiation successful
0621 07:50:58.993964 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":63}
I20260621 07:50:58.994127 18289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:58.990088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:39139 (local address 127.0.192.195:43045)
0621 07:50:58.990401 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:58.990414 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:50:58.990506 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:50:58.990756 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:50:58.990758 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:50:58.990774 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:50:58.990866 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:58.990869 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:58.991680 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:58.991683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:58.992791 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:58.992796 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:58.993593 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:58.993597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:58.993750 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:58.993776 (+ 26us) client_negotiation.cc:769] Sending connection context
0621 07:50:58.993852 (+ 76us) client_negotiation.cc:240] Negotiation successful
0621 07:50:58.993969 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":74,"thread_start_us":118,"threads_started":1}
I20260621 07:50:58.994781 18083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "104b7473444e49b884e663bf131fbf01" candidate_uuid: "1e0a827ef2884d8a81ad2bf9ca9f6cea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f724e6df87394f438fae8a4682986b5e" is_pre_election: true
I20260621 07:50:58.994865 17949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "104b7473444e49b884e663bf131fbf01" candidate_uuid: "1e0a827ef2884d8a81ad2bf9ca9f6cea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd0294f749504ea883195192c19487a9" is_pre_election: true
I20260621 07:50:58.995095 18083 raft_consensus.cc:2454] T 104b7473444e49b884e663bf131fbf01 P f724e6df87394f438fae8a4682986b5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e0a827ef2884d8a81ad2bf9ca9f6cea in term 0.
I20260621 07:50:58.995152 17949 raft_consensus.cc:2454] T 104b7473444e49b884e663bf131fbf01 P cd0294f749504ea883195192c19487a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e0a827ef2884d8a81ad2bf9ca9f6cea in term 0.
I20260621 07:50:58.995563 18151 leader_election.cc:302] T 104b7473444e49b884e663bf131fbf01 P 1e0a827ef2884d8a81ad2bf9ca9f6cea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e0a827ef2884d8a81ad2bf9ca9f6cea, f724e6df87394f438fae8a4682986b5e; no voters:
I20260621 07:50:58.995851 18287 raft_consensus.cc:2781] T 104b7473444e49b884e663bf131fbf01 P 1e0a827ef2884d8a81ad2bf9ca9f6cea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:50:58.995932 18287 raft_consensus.cc:482] T 104b7473444e49b884e663bf131fbf01 P 1e0a827ef2884d8a81ad2bf9ca9f6cea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:50:58.995962 18287 raft_consensus.cc:3037] T 104b7473444e49b884e663bf131fbf01 P 1e0a827ef2884d8a81ad2bf9ca9f6cea [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:50:58.997049 18287 raft_consensus.cc:504] T 104b7473444e49b884e663bf131fbf01 P 1e0a827ef2884d8a81ad2bf9ca9f6cea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f724e6df87394f438fae8a4682986b5e" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 45103 } } peers { permanent_uuid: "1e0a827ef2884d8a81ad2bf9ca9f6cea" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44153 } } peers { permanent_uuid: "cd0294f749504ea883195192c19487a9" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 39139 } }
I20260621 07:50:58.997351 18287 leader_election.cc:288] T 104b7473444e49b884e663bf131fbf01 P 1e0a827ef2884d8a81ad2bf9ca9f6cea [CANDIDATE]: Term 1 election: Requested vote from peers f724e6df87394f438fae8a4682986b5e (127.0.192.194:45103), cd0294f749504ea883195192c19487a9 (127.0.192.193:39139)
I20260621 07:50:58.997685 18083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "104b7473444e49b884e663bf131fbf01" candidate_uuid: "1e0a827ef2884d8a81ad2bf9ca9f6cea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f724e6df87394f438fae8a4682986b5e"
I20260621 07:50:58.997748 17949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "104b7473444e49b884e663bf131fbf01" candidate_uuid: "1e0a827ef2884d8a81ad2bf9ca9f6cea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd0294f749504ea883195192c19487a9"
I20260621 07:50:58.997831 18083 raft_consensus.cc:3037] T 104b7473444e49b884e663bf131fbf01 P f724e6df87394f438fae8a4682986b5e [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:50:58.997884 17949 raft_consensus.cc:3037] T 104b7473444e49b884e663bf131fbf01 P cd0294f749504ea883195192c19487a9 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:50:58.998764 18083 raft_consensus.cc:2454] T 104b7473444e49b884e663bf131fbf01 P f724e6df87394f438fae8a4682986b5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e0a827ef2884d8a81ad2bf9ca9f6cea in term 1.
I20260621 07:50:58.998843 17949 raft_consensus.cc:2454] T 104b7473444e49b884e663bf131fbf01 P cd0294f749504ea883195192c19487a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e0a827ef2884d8a81ad2bf9ca9f6cea in term 1.
I20260621 07:50:58.999078 18151 leader_election.cc:302] T 104b7473444e49b884e663bf131fbf01 P 1e0a827ef2884d8a81ad2bf9ca9f6cea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e0a827ef2884d8a81ad2bf9ca9f6cea, f724e6df87394f438fae8a4682986b5e; no voters:
I20260621 07:50:58.999250 18287 raft_consensus.cc:2781] T 104b7473444e49b884e663bf131fbf01 P 1e0a827ef2884d8a81ad2bf9ca9f6cea [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:50:58.999485 18287 raft_consensus.cc:686] T 104b7473444e49b884e663bf131fbf01 P 1e0a827ef2884d8a81ad2bf9ca9f6cea [term 1 LEADER]: Becoming Leader. State: Replica: 1e0a827ef2884d8a81ad2bf9ca9f6cea, State: Running, Role: LEADER
I20260621 07:50:58.999861 18287 consensus_queue.cc:237] T 104b7473444e49b884e663bf131fbf01 P 1e0a827ef2884d8a81ad2bf9ca9f6cea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f724e6df87394f438fae8a4682986b5e" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 45103 } } peers { permanent_uuid: "1e0a827ef2884d8a81ad2bf9ca9f6cea" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44153 } } peers { permanent_uuid: "cd0294f749504ea883195192c19487a9" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 39139 } }
I20260621 07:50:59.002875 17802 catalog_manager.cc:5495] T 104b7473444e49b884e663bf131fbf01 P 1e0a827ef2884d8a81ad2bf9ca9f6cea reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e0a827ef2884d8a81ad2bf9ca9f6cea (127.0.192.195). New cstate: current_term: 1 leader_uuid: "1e0a827ef2884d8a81ad2bf9ca9f6cea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f724e6df87394f438fae8a4682986b5e" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 45103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e0a827ef2884d8a81ad2bf9ca9f6cea" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 44153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd0294f749504ea883195192c19487a9" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 39139 } health_report { overall_health: UNKNOWN } } }
W20260621 07:50:59.033449 18264 tablet.cc:2369] T 104b7473444e49b884e663bf131fbf01 P 1e0a827ef2884d8a81ad2bf9ca9f6cea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:50:59.054747 18277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:59.051711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52948 (local address 127.0.192.195:44153)
0621 07:50:59.051844 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0621 07:50:59.051849 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:50:59.051868 (+ 19us) server_negotiation.cc:407] Connection header received
0621 07:50:59.051929 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:50:59.051931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:50:59.051979 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:50:59.052059 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:50:59.052270 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:59.052751 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:59.053509 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:59.053698 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:59.053767 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:59.054003 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:50:59.054369 (+ 366us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:50:59.054467 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0621 07:50:59.054525 (+ 58us) server_negotiation.cc:299] Negotiation successful
0621 07:50:59.054567 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 07:50:59.060879 17949 raft_consensus.cc:1261] T 104b7473444e49b884e663bf131fbf01 P cd0294f749504ea883195192c19487a9 [term 1 FOLLOWER]: Refusing update from remote peer 1e0a827ef2884d8a81ad2bf9ca9f6cea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:50:59.060949 18083 raft_consensus.cc:1261] T 104b7473444e49b884e663bf131fbf01 P f724e6df87394f438fae8a4682986b5e [term 1 FOLLOWER]: Refusing update from remote peer 1e0a827ef2884d8a81ad2bf9ca9f6cea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:50:59.061604 18287 consensus_queue.cc:1035] T 104b7473444e49b884e663bf131fbf01 P 1e0a827ef2884d8a81ad2bf9ca9f6cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd0294f749504ea883195192c19487a9" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 39139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:50:59.061873 18290 consensus_queue.cc:1035] T 104b7473444e49b884e663bf131fbf01 P 1e0a827ef2884d8a81ad2bf9ca9f6cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "f724e6df87394f438fae8a4682986b5e" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 45103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:50:59.067342 18293 mvcc.cc:205] Tried to move back new op lower bound from 7299187749104402432 to 7299187748866334720. Current Snapshot: MvccSnapshot[applied={T|T < 7299187749104402432}]
I20260621 07:50:59.067885 18295 mvcc.cc:205] Tried to move back new op lower bound from 7299187749104402432 to 7299187748866334720. Current Snapshot: MvccSnapshot[applied={T|T < 7299187749104402432}]
I20260621 07:50:59.071211 18294 mvcc.cc:205] Tried to move back new op lower bound from 7299187749104402432 to 7299187748866334720. Current Snapshot: MvccSnapshot[applied={T|T < 7299187749104402432}]
I20260621 07:50:59.075305 17802 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:50:59.075440 17802 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:50:59.077387 17802 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 79e920e4a3fd4a11814bc75cfdfd7c2e: Sending DeleteTablet for 3 replicas of tablet 104b7473444e49b884e663bf131fbf01
I20260621 07:50:59.077955 18063 tablet_service.cc:1514] Processing DeleteTablet for tablet 104b7473444e49b884e663bf131fbf01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:50:59 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:39894
I20260621 07:50:59.078095 18197 tablet_service.cc:1514] Processing DeleteTablet for tablet 104b7473444e49b884e663bf131fbf01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:50:59 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:52932
I20260621 07:50:59.078178 17929 tablet_service.cc:1514] Processing DeleteTablet for tablet 104b7473444e49b884e663bf131fbf01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:50:59 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:45574
I20260621 07:50:59.078545 18305 tablet_replica.cc:330] T 104b7473444e49b884e663bf131fbf01 P cd0294f749504ea883195192c19487a9: stopping tablet replica
I20260621 07:50:59.078632 18303 tablet_replica.cc:330] T 104b7473444e49b884e663bf131fbf01 P f724e6df87394f438fae8a4682986b5e: stopping tablet replica
I20260621 07:50:59.078867 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 17866
I20260621 07:50:59.078863 18305 raft_consensus.cc:2229] T 104b7473444e49b884e663bf131fbf01 P cd0294f749504ea883195192c19487a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:50:59.078945 18303 raft_consensus.cc:2229] T 104b7473444e49b884e663bf131fbf01 P f724e6df87394f438fae8a4682986b5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:50:59.079118 18305 raft_consensus.cc:2258] T 104b7473444e49b884e663bf131fbf01 P cd0294f749504ea883195192c19487a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:50:59.079180 18303 raft_consensus.cc:2258] T 104b7473444e49b884e663bf131fbf01 P f724e6df87394f438fae8a4682986b5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:50:59.081539 18303 ts_tablet_manager.cc:1905] T 104b7473444e49b884e663bf131fbf01 P f724e6df87394f438fae8a4682986b5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:50:59.084527 18303 ts_tablet_manager.cc:1918] T 104b7473444e49b884e663bf131fbf01 P f724e6df87394f438fae8a4682986b5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:50:59.084640 18303 log.cc:1196] T 104b7473444e49b884e663bf131fbf01 P f724e6df87394f438fae8a4682986b5e: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-1/wal/wals/104b7473444e49b884e663bf131fbf01
I20260621 07:50:59.084969 18303 ts_tablet_manager.cc:1939] T 104b7473444e49b884e663bf131fbf01 P f724e6df87394f438fae8a4682986b5e: Deleting consensus metadata
I20260621 07:50:59.085438 18304 tablet_replica.cc:330] T 104b7473444e49b884e663bf131fbf01 P 1e0a827ef2884d8a81ad2bf9ca9f6cea: stopping tablet replica
I20260621 07:50:59.085665 18304 raft_consensus.cc:2229] T 104b7473444e49b884e663bf131fbf01 P 1e0a827ef2884d8a81ad2bf9ca9f6cea [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:50:59.085933 18304 raft_consensus.cc:2258] T 104b7473444e49b884e663bf131fbf01 P 1e0a827ef2884d8a81ad2bf9ca9f6cea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:50:59.085974 17789 catalog_manager.cc:4841] TS f724e6df87394f438fae8a4682986b5e (127.0.192.194:45103): tablet 104b7473444e49b884e663bf131fbf01 (table test-table [id=9ec5b38166c04a55ac8966c0375c45a8]) successfully deleted
W20260621 07:50:59.086898 17789 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.192.193:39139 (error 108)
I20260621 07:50:59.087221 18304 ts_tablet_manager.cc:1905] T 104b7473444e49b884e663bf131fbf01 P 1e0a827ef2884d8a81ad2bf9ca9f6cea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:50:59.087700 18278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:59.087458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:39139 (local address 127.0.0.1:45578)
0621 07:50:59.087559 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:59.087620 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.193:39139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260621 07:50:59.087797 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 18000
W20260621 07:50:59.089092 17789 catalog_manager.cc:4584] TS cd0294f749504ea883195192c19487a9 (127.0.192.193:39139): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 104b7473444e49b884e663bf131fbf01: Network error: Client connection negotiation failed: client connection to 127.0.192.193:39139: connect: Connection refused (error 111)
I20260621 07:50:59.091476 18304 ts_tablet_manager.cc:1918] T 104b7473444e49b884e663bf131fbf01 P 1e0a827ef2884d8a81ad2bf9ca9f6cea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:50:59.091557 18304 log.cc:1196] T 104b7473444e49b884e663bf131fbf01 P 1e0a827ef2884d8a81ad2bf9ca9f6cea: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782028028522580-771-0/minicluster-data/ts-2/wal/wals/104b7473444e49b884e663bf131fbf01
I20260621 07:50:59.091871 18304 ts_tablet_manager.cc:1939] T 104b7473444e49b884e663bf131fbf01 P 1e0a827ef2884d8a81ad2bf9ca9f6cea: Deleting consensus metadata
I20260621 07:50:59.092909 17789 catalog_manager.cc:4841] TS 1e0a827ef2884d8a81ad2bf9ca9f6cea (127.0.192.195:44153): tablet 104b7473444e49b884e663bf131fbf01 (table test-table [id=9ec5b38166c04a55ac8966c0375c45a8]) successfully deleted
I20260621 07:50:59.092938 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 18134
I20260621 07:50:59.098075 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 17772
2026-06-21T07:50:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2168 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-21T07:50:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:50:59Z Disabled control of system clock
I20260621 07:50:59.125308 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:45773
--webserver_interface=127.0.192.254
--webserver_port=0
--builtin_ntp_servers=127.0.192.212:43899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:45773
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 07:50:59.221311 18310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:50:59.221511 18310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:50:59.221568 18310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:50:59.225277 18310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:50:59.225324 18310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:50:59.225342 18310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:50:59.225387 18310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:50:59.225419 18310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:50:59.230055 18310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:43899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:45773
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:45773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:50:59.230862 18310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:50:59.231921 18310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:50:59.239370 18315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:50:59.239527 18316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:50:59.239348 18318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:50:59.239619 18310 server_base.cc:1034] running on GCE node
I20260621 07:50:59.240267 18310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:50:59.241297 18310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:50:59.242523 18310 hybrid_clock.cc:648] HybridClock initialized: now 1782028259242513 us; error 27 us; skew 500 ppm
I20260621 07:50:59.244460 18310 webserver.cc:460] Webserver started at http://127.0.192.254:37651/ using document root <none> and password file <none>
I20260621 07:50:59.245036 18310 fs_manager.cc:359] Metadata directory not provided
I20260621 07:50:59.245134 18310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:50:59.245375 18310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:50:59.247257 18310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "81ed95aecbe043e0be6d54b169af9fff"
format_stamp: "Formatted at 2026-06-21 07:50:59 on dist-test-slave-lzx3"
server_key: "3660ceb9fc21d46c0b0226b7b1e8905c"
server_key_iv: "166e9f1417a8b1f2afc4bab403ac01e3"
server_key_version: "encryptionkey@0"
I20260621 07:50:59.247802 18310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "81ed95aecbe043e0be6d54b169af9fff"
format_stamp: "Formatted at 2026-06-21 07:50:59 on dist-test-slave-lzx3"
server_key: "3660ceb9fc21d46c0b0226b7b1e8905c"
server_key_iv: "166e9f1417a8b1f2afc4bab403ac01e3"
server_key_version: "encryptionkey@0"
I20260621 07:50:59.251197 18310 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260621 07:50:59.253345 18324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:50:59.254354 18310 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260621 07:50:59.254491 18310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "81ed95aecbe043e0be6d54b169af9fff"
format_stamp: "Formatted at 2026-06-21 07:50:59 on dist-test-slave-lzx3"
server_key: "3660ceb9fc21d46c0b0226b7b1e8905c"
server_key_iv: "166e9f1417a8b1f2afc4bab403ac01e3"
server_key_version: "encryptionkey@0"
I20260621 07:50:59.254588 18310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:50:59.280319 18310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:50:59.280898 18310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:50:59.281075 18310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:50:59.287253 18310 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:45773
I20260621 07:50:59.287276 18376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:45773 every 8 connection(s)
I20260621 07:50:59.288008 18310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:50:59.289975 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 18310
I20260621 07:50:59.290107 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:50:59.290370 771 external_mini_cluster.cc:1428] Setting key 1c4ae493d60bfe4621280c9d9bc2ba76
I20260621 07:50:59.291049 18377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:50:59.297043 18377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81ed95aecbe043e0be6d54b169af9fff: Bootstrap starting.
I20260621 07:50:59.299342 18377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81ed95aecbe043e0be6d54b169af9fff: Neither blocks nor log segments found. Creating new log.
I20260621 07:50:59.299431 18380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:59.291523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36774 (local address 127.0.192.254:45773)
0621 07:50:59.292084 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0621 07:50:59.292102 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0621 07:50:59.292138 (+ 36us) server_negotiation.cc:407] Connection header received
0621 07:50:59.292920 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:50:59.292940 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:50:59.293371 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:50:59.293690 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:50:59.294087 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:59.295383 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:59.296205 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:59.296832 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:59.296970 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:59.297646 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:50:59.297651 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:50:59.297664 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:50:59.298087 (+ 423us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:50:59.298146 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:50:59.298178 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:50:59.298260 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0621 07:50:59.298386 (+ 126us) server_negotiation.cc:299] Negotiation successful
0621 07:50:59.298608 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":87,"threads_started":1}
I20260621 07:50:59.300148 18377 log.cc:824] T 00000000000000000000000000000000 P 81ed95aecbe043e0be6d54b169af9fff: Log is configured to *not* fsync() on all Append() calls
I20260621 07:50:59.301852 18377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81ed95aecbe043e0be6d54b169af9fff: No bootstrap required, opened a new log
I20260621 07:50:59.304939 18377 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81ed95aecbe043e0be6d54b169af9fff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81ed95aecbe043e0be6d54b169af9fff" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45773 } }
I20260621 07:50:59.305109 18377 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81ed95aecbe043e0be6d54b169af9fff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:50:59.305138 18377 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81ed95aecbe043e0be6d54b169af9fff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81ed95aecbe043e0be6d54b169af9fff, State: Initialized, Role: FOLLOWER
I20260621 07:50:59.305478 18377 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81ed95aecbe043e0be6d54b169af9fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81ed95aecbe043e0be6d54b169af9fff" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45773 } }
I20260621 07:50:59.305588 18377 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81ed95aecbe043e0be6d54b169af9fff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:50:59.305629 18377 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81ed95aecbe043e0be6d54b169af9fff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:50:59.305689 18377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81ed95aecbe043e0be6d54b169af9fff [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:50:59.306334 18377 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81ed95aecbe043e0be6d54b169af9fff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81ed95aecbe043e0be6d54b169af9fff" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45773 } }
I20260621 07:50:59.306588 18377 leader_election.cc:302] T 00000000000000000000000000000000 P 81ed95aecbe043e0be6d54b169af9fff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81ed95aecbe043e0be6d54b169af9fff; no voters:
I20260621 07:50:59.306869 18377 leader_election.cc:288] T 00000000000000000000000000000000 P 81ed95aecbe043e0be6d54b169af9fff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:50:59.306967 18382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81ed95aecbe043e0be6d54b169af9fff [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:50:59.307209 18382 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81ed95aecbe043e0be6d54b169af9fff [term 1 LEADER]: Becoming Leader. State: Replica: 81ed95aecbe043e0be6d54b169af9fff, State: Running, Role: LEADER
I20260621 07:50:59.307528 18382 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81ed95aecbe043e0be6d54b169af9fff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81ed95aecbe043e0be6d54b169af9fff" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45773 } }
I20260621 07:50:59.307909 18377 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81ed95aecbe043e0be6d54b169af9fff [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:50:59.309118 18384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81ed95aecbe043e0be6d54b169af9fff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81ed95aecbe043e0be6d54b169af9fff. Latest consensus state: current_term: 1 leader_uuid: "81ed95aecbe043e0be6d54b169af9fff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81ed95aecbe043e0be6d54b169af9fff" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45773 } } }
I20260621 07:50:59.309154 18383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81ed95aecbe043e0be6d54b169af9fff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81ed95aecbe043e0be6d54b169af9fff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81ed95aecbe043e0be6d54b169af9fff" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 45773 } } }
I20260621 07:50:59.309252 18384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81ed95aecbe043e0be6d54b169af9fff [sys.catalog]: This master's current role is: LEADER
I20260621 07:50:59.309252 18383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81ed95aecbe043e0be6d54b169af9fff [sys.catalog]: This master's current role is: LEADER
I20260621 07:50:59.309589 18391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:50:59.311862 18391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:50:59.316231 18391 catalog_manager.cc:1347] Generated new cluster ID: 3708f75e4df34852aef513ea15aaae6f
I20260621 07:50:59.316285 18391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:50:59.321192 18391 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:50:59.321859 18391 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:50:59.326203 18391 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81ed95aecbe043e0be6d54b169af9fff: Generated new TSK 0
I20260621 07:50:59.326749 18391 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:50:59.330459 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.193:0
--local_ip_for_outbound_sockets=127.0.192.193
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:45773
--builtin_ntp_servers=127.0.192.212:43899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 07:50:59.427060 18401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:50:59.427275 18401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:50:59.427325 18401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:50:59.430990 18401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:50:59.431037 18401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:50:59.431094 18401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.193
I20260621 07:50:59.435328 18401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:43899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:45773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.193
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:50:59.436221 18401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:50:59.437536 18401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:50:59.446422 18406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:50:59.446384 18409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:50:59.446396 18407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:50:59.446681 18401 server_base.cc:1034] running on GCE node
I20260621 07:50:59.447033 18401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:50:59.447582 18401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:50:59.448745 18401 hybrid_clock.cc:648] HybridClock initialized: now 1782028259448731 us; error 24 us; skew 500 ppm
I20260621 07:50:59.450445 18401 webserver.cc:460] Webserver started at http://127.0.192.193:43749/ using document root <none> and password file <none>
I20260621 07:50:59.450979 18401 fs_manager.cc:359] Metadata directory not provided
I20260621 07:50:59.451038 18401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:50:59.451264 18401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:50:59.453162 18401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-0/data/instance:
uuid: "e823f8698d18455a8a37c4c7fa8ac561"
format_stamp: "Formatted at 2026-06-21 07:50:59 on dist-test-slave-lzx3"
server_key: "f8ec5bc3630ce568fe7b9134c82ec1e7"
server_key_iv: "9ea933f1f18d8c51c08fea729fa934a0"
server_key_version: "encryptionkey@0"
I20260621 07:50:59.453696 18401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-0/wal/instance:
uuid: "e823f8698d18455a8a37c4c7fa8ac561"
format_stamp: "Formatted at 2026-06-21 07:50:59 on dist-test-slave-lzx3"
server_key: "f8ec5bc3630ce568fe7b9134c82ec1e7"
server_key_iv: "9ea933f1f18d8c51c08fea729fa934a0"
server_key_version: "encryptionkey@0"
I20260621 07:50:59.456880 18401 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:50:59.458998 18415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:50:59.460085 18401 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:50:59.460196 18401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-0/wal
uuid: "e823f8698d18455a8a37c4c7fa8ac561"
format_stamp: "Formatted at 2026-06-21 07:50:59 on dist-test-slave-lzx3"
server_key: "f8ec5bc3630ce568fe7b9134c82ec1e7"
server_key_iv: "9ea933f1f18d8c51c08fea729fa934a0"
server_key_version: "encryptionkey@0"
I20260621 07:50:59.460284 18401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:50:59.477006 18401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:50:59.477520 18401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:50:59.477695 18401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:50:59.478296 18401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:50:59.479423 18401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:50:59.479468 18401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:50:59.479528 18401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:50:59.479583 18401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:50:59.487675 18401 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.193:42093
I20260621 07:50:59.487711 18528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.193:42093 every 8 connection(s)
I20260621 07:50:59.488438 18401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
I20260621 07:50:59.495749 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 18401
I20260621 07:50:59.495841 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-0/wal/instance
I20260621 07:50:59.496129 771 external_mini_cluster.cc:1428] Setting key d2c671e94926cf42d451bb1ee204ebcd
I20260621 07:50:59.496443 18380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:59.489988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:47107 (local address 127.0.192.254:45773)
0621 07:50:59.490108 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0621 07:50:59.490112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:50:59.490957 (+ 845us) server_negotiation.cc:407] Connection header received
0621 07:50:59.492208 (+ 1251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:50:59.492211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:50:59.492271 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:50:59.492345 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:50:59.493195 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:59.493694 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:59.494680 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:59.494895 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:59.494997 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:59.495981 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:50:59.495985 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:50:59.495987 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:50:59.496004 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:50:59.496022 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:50:59.496029 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:50:59.496096 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0621 07:50:59.496260 (+ 164us) server_negotiation.cc:299] Negotiation successful
0621 07:50:59.496319 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260621 07:50:59.497176 18531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:59.490295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45773 (local address 127.0.192.193:47107)
0621 07:50:59.490825 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:59.490858 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0621 07:50:59.492008 (+ 1150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:50:59.492492 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:50:59.492498 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:50:59.492644 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:50:59.493049 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:59.493060 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:59.493807 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:59.493810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:59.494564 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:59.494573 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:59.495023 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:59.495026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:59.495180 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:59.495686 (+ 506us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:50:59.495688 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:50:59.495719 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:50:59.495770 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:50:59.496118 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:50:59.496126 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:50:59.496132 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 07:50:59.496259 (+ 127us) client_negotiation.cc:240] Negotiation successful
0621 07:50:59.496479 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":102,"threads_started":1}
I20260621 07:50:59.497888 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.194:0
--local_ip_for_outbound_sockets=127.0.192.194
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:45773
--builtin_ntp_servers=127.0.192.212:43899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 07:50:59.498241 18529 heartbeater.cc:347] Connected to a master server at 127.0.192.254:45773
I20260621 07:50:59.498446 18529 heartbeater.cc:464] Registering TS with master...
I20260621 07:50:59.498970 18529 heartbeater.cc:511] Master 127.0.192.254:45773 requested a full tablet report, sending...
I20260621 07:50:59.500481 18341 ts_manager.cc:194] Registered new tserver with Master: e823f8698d18455a8a37c4c7fa8ac561 (127.0.192.193:42093)
I20260621 07:50:59.502010 18341 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.192.193:47107
W20260621 07:50:59.599283 18532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:50:59.599498 18532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:50:59.599532 18532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:50:59.603273 18532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:50:59.603312 18532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:50:59.603360 18532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.194
I20260621 07:50:59.607795 18532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:43899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:45773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.194
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:50:59.608760 18532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:50:59.610006 18532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:50:59.618521 18537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:50:59.618616 18540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:50:59.618705 18538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:50:59.618767 18532 server_base.cc:1034] running on GCE node
I20260621 07:50:59.619225 18532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:50:59.619776 18532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:50:59.620965 18532 hybrid_clock.cc:648] HybridClock initialized: now 1782028259620935 us; error 42 us; skew 500 ppm
I20260621 07:50:59.622596 18532 webserver.cc:460] Webserver started at http://127.0.192.194:40163/ using document root <none> and password file <none>
I20260621 07:50:59.623171 18532 fs_manager.cc:359] Metadata directory not provided
I20260621 07:50:59.623273 18532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:50:59.623483 18532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:50:59.625342 18532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-1/data/instance:
uuid: "c156aeefaa3045ef9987772ceb86b832"
format_stamp: "Formatted at 2026-06-21 07:50:59 on dist-test-slave-lzx3"
server_key: "d83b1fb69cc3ef8c8b642f27ec8a8b90"
server_key_iv: "619aa13f75846658f0b02aac74ba6a11"
server_key_version: "encryptionkey@0"
I20260621 07:50:59.625830 18532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-1/wal/instance:
uuid: "c156aeefaa3045ef9987772ceb86b832"
format_stamp: "Formatted at 2026-06-21 07:50:59 on dist-test-slave-lzx3"
server_key: "d83b1fb69cc3ef8c8b642f27ec8a8b90"
server_key_iv: "619aa13f75846658f0b02aac74ba6a11"
server_key_version: "encryptionkey@0"
I20260621 07:50:59.629017 18532 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:50:59.631036 18546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:50:59.631986 18532 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:50:59.632114 18532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-1/wal
uuid: "c156aeefaa3045ef9987772ceb86b832"
format_stamp: "Formatted at 2026-06-21 07:50:59 on dist-test-slave-lzx3"
server_key: "d83b1fb69cc3ef8c8b642f27ec8a8b90"
server_key_iv: "619aa13f75846658f0b02aac74ba6a11"
server_key_version: "encryptionkey@0"
I20260621 07:50:59.632211 18532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:50:59.671746 18532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:50:59.672595 18532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:50:59.672785 18532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:50:59.673506 18532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:50:59.674817 18532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:50:59.674904 18532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:50:59.674994 18532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:50:59.675033 18532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:50:59.684303 18532 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.194:35855
I20260621 07:50:59.684327 18659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.194:35855 every 8 connection(s)
I20260621 07:50:59.685191 18532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
I20260621 07:50:59.693943 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 18532
I20260621 07:50:59.694070 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-1/wal/instance
I20260621 07:50:59.694331 771 external_mini_cluster.cc:1428] Setting key f211359cb6e9c5a6a14e050dc6a0a1ba
I20260621 07:50:59.696182 18380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:59.686891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:37669 (local address 127.0.192.254:45773)
0621 07:50:59.687025 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0621 07:50:59.687029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:50:59.687975 (+ 946us) server_negotiation.cc:407] Connection header received
0621 07:50:59.689363 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:50:59.689366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:50:59.689411 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:50:59.689483 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:50:59.690282 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:59.690761 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:59.691729 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:59.691912 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:59.691979 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:59.695694 (+ 3715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:50:59.695698 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:50:59.695700 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:50:59.695712 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:50:59.695732 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:50:59.695741 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:50:59.695816 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0621 07:50:59.695939 (+ 123us) server_negotiation.cc:299] Negotiation successful
0621 07:50:59.696012 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 07:50:59.696192 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.195:0
--local_ip_for_outbound_sockets=127.0.192.195
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:45773
--builtin_ntp_servers=127.0.192.212:43899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 07:50:59.697018 18662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:59.687254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45773 (local address 127.0.192.194:37669)
0621 07:50:59.687856 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:59.687891 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 07:50:59.689184 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:50:59.689634 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:50:59.689640 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:50:59.689718 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:50:59.690171 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:59.690182 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:59.690853 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:59.690856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:59.691636 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:59.691644 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:59.692025 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:59.692027 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:59.692150 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:59.695462 (+ 3312us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:50:59.695464 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:50:59.695513 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:50:59.695568 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:50:59.695822 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:50:59.695827 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:50:59.695831 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 07:50:59.695912 (+ 81us) client_negotiation.cc:240] Negotiation successful
0621 07:50:59.696097 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":105,"threads_started":1}
I20260621 07:50:59.698122 18660 heartbeater.cc:347] Connected to a master server at 127.0.192.254:45773
I20260621 07:50:59.698323 18660 heartbeater.cc:464] Registering TS with master...
I20260621 07:50:59.698814 18660 heartbeater.cc:511] Master 127.0.192.254:45773 requested a full tablet report, sending...
I20260621 07:50:59.699757 18341 ts_manager.cc:194] Registered new tserver with Master: c156aeefaa3045ef9987772ceb86b832 (127.0.192.194:35855)
I20260621 07:50:59.700268 18341 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.192.194:37669
W20260621 07:50:59.796551 18663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:50:59.796762 18663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:50:59.796792 18663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:50:59.800422 18663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:50:59.800462 18663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:50:59.800526 18663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.195
I20260621 07:50:59.804838 18663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:43899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:45773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.195
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:50:59.805610 18663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:50:59.806660 18663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:50:59.815157 18671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:50:59.815220 18669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:50:59.815172 18668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:50:59.815852 18663 server_base.cc:1034] running on GCE node
I20260621 07:50:59.816203 18663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:50:59.816808 18663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:50:59.817982 18663 hybrid_clock.cc:648] HybridClock initialized: now 1782028259817973 us; error 23 us; skew 500 ppm
I20260621 07:50:59.819674 18663 webserver.cc:460] Webserver started at http://127.0.192.195:35359/ using document root <none> and password file <none>
I20260621 07:50:59.820187 18663 fs_manager.cc:359] Metadata directory not provided
I20260621 07:50:59.820268 18663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:50:59.820472 18663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:50:59.822193 18663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-2/data/instance:
uuid: "c19af2187aa34f0ba0eb85d263a05e78"
format_stamp: "Formatted at 2026-06-21 07:50:59 on dist-test-slave-lzx3"
server_key: "d01e4c12666f9c4d575c03eb49e511f4"
server_key_iv: "17c13329d0caaa34062db59a64b0a493"
server_key_version: "encryptionkey@0"
I20260621 07:50:59.822687 18663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-2/wal/instance:
uuid: "c19af2187aa34f0ba0eb85d263a05e78"
format_stamp: "Formatted at 2026-06-21 07:50:59 on dist-test-slave-lzx3"
server_key: "d01e4c12666f9c4d575c03eb49e511f4"
server_key_iv: "17c13329d0caaa34062db59a64b0a493"
server_key_version: "encryptionkey@0"
I20260621 07:50:59.825834 18663 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:50:59.828171 18677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:50:59.829309 18663 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260621 07:50:59.829443 18663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-2/wal
uuid: "c19af2187aa34f0ba0eb85d263a05e78"
format_stamp: "Formatted at 2026-06-21 07:50:59 on dist-test-slave-lzx3"
server_key: "d01e4c12666f9c4d575c03eb49e511f4"
server_key_iv: "17c13329d0caaa34062db59a64b0a493"
server_key_version: "encryptionkey@0"
I20260621 07:50:59.829546 18663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:50:59.854794 18663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:50:59.855420 18663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:50:59.855594 18663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:50:59.856142 18663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:50:59.857030 18663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:50:59.857098 18663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:50:59.857158 18663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:50:59.857205 18663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:50:59.865531 18663 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.195:38741
I20260621 07:50:59.865556 18790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.195:38741 every 8 connection(s)
I20260621 07:50:59.866330 18663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
I20260621 07:50:59.871925 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 18663
I20260621 07:50:59.872053 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-2/wal/instance
I20260621 07:50:59.872327 771 external_mini_cluster.cc:1428] Setting key fa3466384c45b6677d7629c163cf3bde
I20260621 07:50:59.874879 18380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:59.867961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:35785 (local address 127.0.192.254:45773)
0621 07:50:59.868119 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0621 07:50:59.868123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:50:59.869153 (+ 1030us) server_negotiation.cc:407] Connection header received
0621 07:50:59.870343 (+ 1190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:50:59.870346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:50:59.870391 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:50:59.870468 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:50:59.871207 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:59.871670 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:59.873019 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:59.873248 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:59.873370 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:59.874437 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:50:59.874440 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:50:59.874443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:50:59.874455 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:50:59.874475 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:50:59.874489 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:50:59.874558 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0621 07:50:59.874706 (+ 148us) server_negotiation.cc:299] Negotiation successful
0621 07:50:59.874758 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260621 07:50:59.875602 18793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:59.868339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:45773 (local address 127.0.192.195:35785)
0621 07:50:59.868986 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:59.869024 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 07:50:59.870174 (+ 1150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:50:59.870611 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:50:59.870616 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:50:59.870686 (+ 70us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:50:59.871091 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:59.871101 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:59.871785 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:59.871789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:59.872902 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:59.872910 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:59.873405 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:59.873408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:59.873591 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:59.874186 (+ 595us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:50:59.874188 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:50:59.874232 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:50:59.874307 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:50:59.874571 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:50:59.874579 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:50:59.874583 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 07:50:59.874693 (+ 110us) client_negotiation.cc:240] Negotiation successful
0621 07:50:59.874925 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":106,"threads_started":1}
I20260621 07:50:59.876566 18791 heartbeater.cc:347] Connected to a master server at 127.0.192.254:45773
I20260621 07:50:59.876778 18791 heartbeater.cc:464] Registering TS with master...
I20260621 07:50:59.877226 18791 heartbeater.cc:511] Master 127.0.192.254:45773 requested a full tablet report, sending...
I20260621 07:50:59.878137 18341 ts_manager.cc:194] Registered new tserver with Master: c19af2187aa34f0ba0eb85d263a05e78 (127.0.192.195:38741)
I20260621 07:50:59.878664 18341 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.192.195:35785
I20260621 07:50:59.885871 771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:50:59.890827 18380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:59.887575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58710 (local address 127.0.192.254:45773)
0621 07:50:59.887692 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0621 07:50:59.887698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:50:59.887816 (+ 118us) server_negotiation.cc:407] Connection header received
0621 07:50:59.887902 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:50:59.887904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:50:59.887936 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:50:59.887994 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:50:59.888245 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:59.888784 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:59.889550 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:59.889759 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:59.889825 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:59.890488 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:50:59.890491 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:50:59.890492 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:50:59.890501 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:50:59.890516 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:50:59.890523 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:50:59.890571 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0621 07:50:59.890650 (+ 79us) server_negotiation.cc:299] Negotiation successful
0621 07:50:59.890686 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 07:50:59.892906 18341 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58710:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:50:59.895287 18341 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:50:59.907459 18804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:59.902480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:35855 (local address 127.0.0.1:50600)
0621 07:50:59.902765 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:59.902776 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 07:50:59.902855 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:50:59.903687 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:50:59.903692 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:50:59.903710 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:50:59.903813 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:59.903819 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:59.904827 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:59.904832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:59.905938 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:59.905949 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:59.906938 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:59.906942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:59.907048 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:59.907081 (+ 33us) client_negotiation.cc:769] Sending connection context
0621 07:50:59.907195 (+ 114us) client_negotiation.cc:240] Negotiation successful
0621 07:50:59.907323 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":98,"threads_started":1}
I20260621 07:50:59.907500 18802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:59.902065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.195:38741 (local address 127.0.0.1:40460)
0621 07:50:59.902347 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:59.902396 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0621 07:50:59.902504 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:50:59.903826 (+ 1322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:50:59.903828 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:50:59.903845 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:50:59.904023 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:59.904030 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:59.905324 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:59.905327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:59.906325 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:59.906333 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:59.907197 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:59.907201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:59.907271 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:59.907291 (+ 20us) client_negotiation.cc:769] Sending connection context
0621 07:50:59.907350 (+ 59us) client_negotiation.cc:240] Negotiation successful
0621 07:50:59.907399 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":139,"thread_start_us":63,"threads_started":1}
I20260621 07:50:59.907595 18803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:59.902138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40460 (local address 127.0.192.195:38741)
0621 07:50:59.902494 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0621 07:50:59.902504 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:50:59.902525 (+ 21us) server_negotiation.cc:407] Connection header received
0621 07:50:59.902634 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:50:59.902639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:50:59.903029 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:50:59.903156 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:50:59.904148 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:59.905180 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:59.906433 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:59.907102 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:59.907194 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:59.907332 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0621 07:50:59.907378 (+ 46us) server_negotiation.cc:299] Negotiation successful
0621 07:50:59.907455 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":89,"threads_started":1}
I20260621 07:50:59.907773 18806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:59.902781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:42093 (local address 127.0.0.1:60938)
0621 07:50:59.903053 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0621 07:50:59.903064 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 07:50:59.903132 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:50:59.903800 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:50:59.903803 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:50:59.903819 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:50:59.903998 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:59.904003 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:59.905298 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:59.905301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:59.906532 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:50:59.906542 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:50:59.907436 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:50:59.907439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:50:59.907528 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:59.907547 (+ 19us) client_negotiation.cc:769] Sending connection context
0621 07:50:59.907592 (+ 45us) client_negotiation.cc:240] Negotiation successful
0621 07:50:59.907654 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":131,"spinlock_wait_cycles":896,"thread_start_us":76,"threads_started":1}
I20260621 07:50:59.908372 18805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:59.902717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50600 (local address 127.0.192.194:35855)
0621 07:50:59.903345 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0621 07:50:59.903357 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:50:59.903379 (+ 22us) server_negotiation.cc:407] Connection header received
0621 07:50:59.903448 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:50:59.903451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:50:59.903561 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:50:59.903692 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:50:59.903926 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:59.904735 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:59.906097 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:59.906815 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:59.906939 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:59.907124 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0621 07:50:59.908077 (+ 953us) server_negotiation.cc:299] Negotiation successful
0621 07:50:59.908237 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":84,"threads_started":1}
I20260621 07:50:59.909067 18807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:50:59.903144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60938 (local address 127.0.192.193:42093)
0621 07:50:59.903488 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0621 07:50:59.903496 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:50:59.903512 (+ 16us) server_negotiation.cc:407] Connection header received
0621 07:50:59.903565 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:50:59.903568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:50:59.903684 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:50:59.903817 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:50:59.904104 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:59.905180 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:59.906653 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:50:59.907320 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:50:59.908587 (+ 1267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:50:59.908766 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0621 07:50:59.908815 (+ 49us) server_negotiation.cc:299] Negotiation successful
0621 07:50:59.908899 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":88,"threads_started":1}
I20260621 07:50:59.910743 18725 tablet_service.cc:1467] Processing CreateTablet for tablet f432687a62334f4e9264c145737c00d2 (DEFAULT_TABLE table=test-table [id=899cb7cd27bc4ddc8c4ef1db433cb273]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:50:59.911194 18463 tablet_service.cc:1467] Processing CreateTablet for tablet f432687a62334f4e9264c145737c00d2 (DEFAULT_TABLE table=test-table [id=899cb7cd27bc4ddc8c4ef1db433cb273]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:50:59.911175 18594 tablet_service.cc:1467] Processing CreateTablet for tablet f432687a62334f4e9264c145737c00d2 (DEFAULT_TABLE table=test-table [id=899cb7cd27bc4ddc8c4ef1db433cb273]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:50:59.911785 18725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f432687a62334f4e9264c145737c00d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:50:59.912109 18463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f432687a62334f4e9264c145737c00d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:50:59.912117 18594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f432687a62334f4e9264c145737c00d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:50:59.917423 18808 tablet_bootstrap.cc:492] T f432687a62334f4e9264c145737c00d2 P e823f8698d18455a8a37c4c7fa8ac561: Bootstrap starting.
I20260621 07:50:59.918900 18810 tablet_bootstrap.cc:492] T f432687a62334f4e9264c145737c00d2 P c156aeefaa3045ef9987772ceb86b832: Bootstrap starting.
I20260621 07:50:59.919163 18809 tablet_bootstrap.cc:492] T f432687a62334f4e9264c145737c00d2 P c19af2187aa34f0ba0eb85d263a05e78: Bootstrap starting.
I20260621 07:50:59.919486 18808 tablet_bootstrap.cc:654] T f432687a62334f4e9264c145737c00d2 P e823f8698d18455a8a37c4c7fa8ac561: Neither blocks nor log segments found. Creating new log.
I20260621 07:50:59.920136 18808 log.cc:824] T f432687a62334f4e9264c145737c00d2 P e823f8698d18455a8a37c4c7fa8ac561: Log is configured to *not* fsync() on all Append() calls
I20260621 07:50:59.920969 18810 tablet_bootstrap.cc:654] T f432687a62334f4e9264c145737c00d2 P c156aeefaa3045ef9987772ceb86b832: Neither blocks nor log segments found. Creating new log.
I20260621 07:50:59.920969 18809 tablet_bootstrap.cc:654] T f432687a62334f4e9264c145737c00d2 P c19af2187aa34f0ba0eb85d263a05e78: Neither blocks nor log segments found. Creating new log.
I20260621 07:50:59.921738 18809 log.cc:824] T f432687a62334f4e9264c145737c00d2 P c19af2187aa34f0ba0eb85d263a05e78: Log is configured to *not* fsync() on all Append() calls
I20260621 07:50:59.921738 18810 log.cc:824] T f432687a62334f4e9264c145737c00d2 P c156aeefaa3045ef9987772ceb86b832: Log is configured to *not* fsync() on all Append() calls
I20260621 07:50:59.921916 18808 tablet_bootstrap.cc:492] T f432687a62334f4e9264c145737c00d2 P e823f8698d18455a8a37c4c7fa8ac561: No bootstrap required, opened a new log
I20260621 07:50:59.922096 18808 ts_tablet_manager.cc:1397] T f432687a62334f4e9264c145737c00d2 P e823f8698d18455a8a37c4c7fa8ac561: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260621 07:50:59.923316 18809 tablet_bootstrap.cc:492] T f432687a62334f4e9264c145737c00d2 P c19af2187aa34f0ba0eb85d263a05e78: No bootstrap required, opened a new log
I20260621 07:50:59.923507 18809 ts_tablet_manager.cc:1397] T f432687a62334f4e9264c145737c00d2 P c19af2187aa34f0ba0eb85d263a05e78: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260621 07:50:59.924082 18810 tablet_bootstrap.cc:492] T f432687a62334f4e9264c145737c00d2 P c156aeefaa3045ef9987772ceb86b832: No bootstrap required, opened a new log
I20260621 07:50:59.924247 18810 ts_tablet_manager.cc:1397] T f432687a62334f4e9264c145737c00d2 P c156aeefaa3045ef9987772ceb86b832: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260621 07:50:59.926023 18808 raft_consensus.cc:348] T f432687a62334f4e9264c145737c00d2 P e823f8698d18455a8a37c4c7fa8ac561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19af2187aa34f0ba0eb85d263a05e78" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 38741 } } peers { permanent_uuid: "c156aeefaa3045ef9987772ceb86b832" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 35855 } } peers { permanent_uuid: "e823f8698d18455a8a37c4c7fa8ac561" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 42093 } }
I20260621 07:50:59.926242 18808 raft_consensus.cc:374] T f432687a62334f4e9264c145737c00d2 P e823f8698d18455a8a37c4c7fa8ac561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:50:59.926316 18808 raft_consensus.cc:729] T f432687a62334f4e9264c145737c00d2 P e823f8698d18455a8a37c4c7fa8ac561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e823f8698d18455a8a37c4c7fa8ac561, State: Initialized, Role: FOLLOWER
I20260621 07:50:59.926169 18809 raft_consensus.cc:348] T f432687a62334f4e9264c145737c00d2 P c19af2187aa34f0ba0eb85d263a05e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19af2187aa34f0ba0eb85d263a05e78" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 38741 } } peers { permanent_uuid: "c156aeefaa3045ef9987772ceb86b832" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 35855 } } peers { permanent_uuid: "e823f8698d18455a8a37c4c7fa8ac561" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 42093 } }
I20260621 07:50:59.926401 18809 raft_consensus.cc:374] T f432687a62334f4e9264c145737c00d2 P c19af2187aa34f0ba0eb85d263a05e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:50:59.926457 18809 raft_consensus.cc:729] T f432687a62334f4e9264c145737c00d2 P c19af2187aa34f0ba0eb85d263a05e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c19af2187aa34f0ba0eb85d263a05e78, State: Initialized, Role: FOLLOWER
I20260621 07:50:59.926708 18808 consensus_queue.cc:260] T f432687a62334f4e9264c145737c00d2 P e823f8698d18455a8a37c4c7fa8ac561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19af2187aa34f0ba0eb85d263a05e78" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 38741 } } peers { permanent_uuid: "c156aeefaa3045ef9987772ceb86b832" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 35855 } } peers { permanent_uuid: "e823f8698d18455a8a37c4c7fa8ac561" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 42093 } }
I20260621 07:50:59.926859 18809 consensus_queue.cc:260] T f432687a62334f4e9264c145737c00d2 P c19af2187aa34f0ba0eb85d263a05e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19af2187aa34f0ba0eb85d263a05e78" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 38741 } } peers { permanent_uuid: "c156aeefaa3045ef9987772ceb86b832" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 35855 } } peers { permanent_uuid: "e823f8698d18455a8a37c4c7fa8ac561" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 42093 } }
I20260621 07:50:59.927295 18529 heartbeater.cc:503] Master 127.0.192.254:45773 was elected leader, sending a full tablet report...
I20260621 07:50:59.927543 18808 ts_tablet_manager.cc:1428] T f432687a62334f4e9264c145737c00d2 P e823f8698d18455a8a37c4c7fa8ac561: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260621 07:50:59.927613 18809 ts_tablet_manager.cc:1428] T f432687a62334f4e9264c145737c00d2 P c19af2187aa34f0ba0eb85d263a05e78: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260621 07:50:59.927850 18791 heartbeater.cc:503] Master 127.0.192.254:45773 was elected leader, sending a full tablet report...
I20260621 07:50:59.928117 18810 raft_consensus.cc:348] T f432687a62334f4e9264c145737c00d2 P c156aeefaa3045ef9987772ceb86b832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19af2187aa34f0ba0eb85d263a05e78" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 38741 } } peers { permanent_uuid: "c156aeefaa3045ef9987772ceb86b832" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 35855 } } peers { permanent_uuid: "e823f8698d18455a8a37c4c7fa8ac561" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 42093 } }
I20260621 07:50:59.928409 18810 raft_consensus.cc:374] T f432687a62334f4e9264c145737c00d2 P c156aeefaa3045ef9987772ceb86b832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:50:59.928501 18810 raft_consensus.cc:729] T f432687a62334f4e9264c145737c00d2 P c156aeefaa3045ef9987772ceb86b832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c156aeefaa3045ef9987772ceb86b832, State: Initialized, Role: FOLLOWER
I20260621 07:50:59.928954 18810 consensus_queue.cc:260] T f432687a62334f4e9264c145737c00d2 P c156aeefaa3045ef9987772ceb86b832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19af2187aa34f0ba0eb85d263a05e78" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 38741 } } peers { permanent_uuid: "c156aeefaa3045ef9987772ceb86b832" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 35855 } } peers { permanent_uuid: "e823f8698d18455a8a37c4c7fa8ac561" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 42093 } }
I20260621 07:50:59.929554 18660 heartbeater.cc:503] Master 127.0.192.254:45773 was elected leader, sending a full tablet report...
I20260621 07:50:59.929842 18810 ts_tablet_manager.cc:1428] T f432687a62334f4e9264c145737c00d2 P c156aeefaa3045ef9987772ceb86b832: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260621 07:50:59.936045 18661 tablet.cc:2369] T f432687a62334f4e9264c145737c00d2 P c156aeefaa3045ef9987772ceb86b832: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 07:50:59.989384 18530 tablet.cc:2369] T f432687a62334f4e9264c145737c00d2 P e823f8698d18455a8a37c4c7fa8ac561: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 07:51:00.117271 18792 tablet.cc:2369] T f432687a62334f4e9264c145737c00d2 P c19af2187aa34f0ba0eb85d263a05e78: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:51:00.245121 18815 raft_consensus.cc:482] T f432687a62334f4e9264c145737c00d2 P c19af2187aa34f0ba0eb85d263a05e78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:51:00.245322 18815 raft_consensus.cc:504] T f432687a62334f4e9264c145737c00d2 P c19af2187aa34f0ba0eb85d263a05e78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19af2187aa34f0ba0eb85d263a05e78" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 38741 } } peers { permanent_uuid: "c156aeefaa3045ef9987772ceb86b832" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 35855 } } peers { permanent_uuid: "e823f8698d18455a8a37c4c7fa8ac561" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 42093 } }
I20260621 07:51:00.246399 18815 leader_election.cc:288] T f432687a62334f4e9264c145737c00d2 P c19af2187aa34f0ba0eb85d263a05e78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c156aeefaa3045ef9987772ceb86b832 (127.0.192.194:35855), e823f8698d18455a8a37c4c7fa8ac561 (127.0.192.193:42093)
I20260621 07:51:00.250214 18793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:00.246587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:35855 (local address 127.0.192.195:37505)
0621 07:51:00.246699 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:00.246731 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:00.246836 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:00.247153 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:00.247156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:00.247177 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:51:00.247270 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:00.247274 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:00.248217 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:00.248220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:00.249150 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:00.249159 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:00.249848 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:00.249851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:00.249936 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:00.249960 (+ 24us) client_negotiation.cc:769] Sending connection context
0621 07:51:00.250029 (+ 69us) client_negotiation.cc:240] Negotiation successful
0621 07:51:00.250091 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":5}
I20260621 07:51:00.250265 18805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:00.246702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:37505 (local address 127.0.192.194:35855)
0621 07:51:00.246864 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:00.246869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:00.246911 (+ 42us) server_negotiation.cc:407] Connection header received
0621 07:51:00.246967 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:00.246970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:00.247019 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:00.247089 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:51:00.247408 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:00.248105 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:00.249275 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:00.249751 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:00.249831 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:00.249863 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:00.250063 (+ 200us) server_negotiation.cc:299] Negotiation successful
0621 07:51:00.250121 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 07:51:00.250458 18817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:00.246888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:42093 (local address 127.0.192.195:40137)
0621 07:51:00.247223 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:00.247233 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:00.247315 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:00.247558 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:00.247560 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:00.247575 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:51:00.247715 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:00.247721 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:00.248515 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:00.248517 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:00.249442 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:00.249452 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:00.250081 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:00.250084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:00.250163 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:00.250184 (+ 21us) client_negotiation.cc:769] Sending connection context
0621 07:51:00.250252 (+ 68us) client_negotiation.cc:240] Negotiation successful
0621 07:51:00.250321 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":20,"thread_start_us":117,"threads_started":1}
I20260621 07:51:00.250464 18807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:00.246986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:40137 (local address 127.0.192.193:42093)
0621 07:51:00.247117 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:00.247122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:00.247311 (+ 189us) server_negotiation.cc:407] Connection header received
0621 07:51:00.247408 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:00.247410 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:00.247446 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:00.247514 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:51:00.247811 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:00.248410 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:00.249567 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:00.249957 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:00.250057 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:00.250091 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:00.250283 (+ 192us) server_negotiation.cc:299] Negotiation successful
0621 07:51:00.250350 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 07:51:00.250960 18614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f432687a62334f4e9264c145737c00d2" candidate_uuid: "c19af2187aa34f0ba0eb85d263a05e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c156aeefaa3045ef9987772ceb86b832" is_pre_election: true
I20260621 07:51:00.251103 18483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f432687a62334f4e9264c145737c00d2" candidate_uuid: "c19af2187aa34f0ba0eb85d263a05e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e823f8698d18455a8a37c4c7fa8ac561" is_pre_election: true
I20260621 07:51:00.251253 18614 raft_consensus.cc:2454] T f432687a62334f4e9264c145737c00d2 P c156aeefaa3045ef9987772ceb86b832 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c19af2187aa34f0ba0eb85d263a05e78 in term 0.
I20260621 07:51:00.251372 18483 raft_consensus.cc:2454] T f432687a62334f4e9264c145737c00d2 P e823f8698d18455a8a37c4c7fa8ac561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c19af2187aa34f0ba0eb85d263a05e78 in term 0.
I20260621 07:51:00.251740 18679 leader_election.cc:302] T f432687a62334f4e9264c145737c00d2 P c19af2187aa34f0ba0eb85d263a05e78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c156aeefaa3045ef9987772ceb86b832, c19af2187aa34f0ba0eb85d263a05e78; no voters:
I20260621 07:51:00.251963 18815 raft_consensus.cc:2781] T f432687a62334f4e9264c145737c00d2 P c19af2187aa34f0ba0eb85d263a05e78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:51:00.252028 18815 raft_consensus.cc:482] T f432687a62334f4e9264c145737c00d2 P c19af2187aa34f0ba0eb85d263a05e78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:51:00.252053 18815 raft_consensus.cc:3037] T f432687a62334f4e9264c145737c00d2 P c19af2187aa34f0ba0eb85d263a05e78 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:00.252955 18815 raft_consensus.cc:504] T f432687a62334f4e9264c145737c00d2 P c19af2187aa34f0ba0eb85d263a05e78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19af2187aa34f0ba0eb85d263a05e78" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 38741 } } peers { permanent_uuid: "c156aeefaa3045ef9987772ceb86b832" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 35855 } } peers { permanent_uuid: "e823f8698d18455a8a37c4c7fa8ac561" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 42093 } }
I20260621 07:51:00.253232 18815 leader_election.cc:288] T f432687a62334f4e9264c145737c00d2 P c19af2187aa34f0ba0eb85d263a05e78 [CANDIDATE]: Term 1 election: Requested vote from peers c156aeefaa3045ef9987772ceb86b832 (127.0.192.194:35855), e823f8698d18455a8a37c4c7fa8ac561 (127.0.192.193:42093)
I20260621 07:51:00.253522 18614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f432687a62334f4e9264c145737c00d2" candidate_uuid: "c19af2187aa34f0ba0eb85d263a05e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c156aeefaa3045ef9987772ceb86b832"
I20260621 07:51:00.253573 18483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f432687a62334f4e9264c145737c00d2" candidate_uuid: "c19af2187aa34f0ba0eb85d263a05e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e823f8698d18455a8a37c4c7fa8ac561"
I20260621 07:51:00.253654 18483 raft_consensus.cc:3037] T f432687a62334f4e9264c145737c00d2 P e823f8698d18455a8a37c4c7fa8ac561 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:00.253654 18614 raft_consensus.cc:3037] T f432687a62334f4e9264c145737c00d2 P c156aeefaa3045ef9987772ceb86b832 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:00.254772 18614 raft_consensus.cc:2454] T f432687a62334f4e9264c145737c00d2 P c156aeefaa3045ef9987772ceb86b832 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c19af2187aa34f0ba0eb85d263a05e78 in term 1.
I20260621 07:51:00.254772 18483 raft_consensus.cc:2454] T f432687a62334f4e9264c145737c00d2 P e823f8698d18455a8a37c4c7fa8ac561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c19af2187aa34f0ba0eb85d263a05e78 in term 1.
I20260621 07:51:00.255015 18679 leader_election.cc:302] T f432687a62334f4e9264c145737c00d2 P c19af2187aa34f0ba0eb85d263a05e78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c156aeefaa3045ef9987772ceb86b832, c19af2187aa34f0ba0eb85d263a05e78; no voters:
I20260621 07:51:00.255144 18815 raft_consensus.cc:2781] T f432687a62334f4e9264c145737c00d2 P c19af2187aa34f0ba0eb85d263a05e78 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:00.255309 18815 raft_consensus.cc:686] T f432687a62334f4e9264c145737c00d2 P c19af2187aa34f0ba0eb85d263a05e78 [term 1 LEADER]: Becoming Leader. State: Replica: c19af2187aa34f0ba0eb85d263a05e78, State: Running, Role: LEADER
I20260621 07:51:00.255589 18815 consensus_queue.cc:237] T f432687a62334f4e9264c145737c00d2 P c19af2187aa34f0ba0eb85d263a05e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19af2187aa34f0ba0eb85d263a05e78" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 38741 } } peers { permanent_uuid: "c156aeefaa3045ef9987772ceb86b832" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 35855 } } peers { permanent_uuid: "e823f8698d18455a8a37c4c7fa8ac561" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 42093 } }
I20260621 07:51:00.258236 18340 catalog_manager.cc:5495] T f432687a62334f4e9264c145737c00d2 P c19af2187aa34f0ba0eb85d263a05e78 reported cstate change: term changed from 0 to 1, leader changed from <none> to c19af2187aa34f0ba0eb85d263a05e78 (127.0.192.195). New cstate: current_term: 1 leader_uuid: "c19af2187aa34f0ba0eb85d263a05e78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19af2187aa34f0ba0eb85d263a05e78" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 38741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c156aeefaa3045ef9987772ceb86b832" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 35855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e823f8698d18455a8a37c4c7fa8ac561" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 42093 } health_report { overall_health: UNKNOWN } } }
I20260621 07:51:00.355908 18803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:00.352585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40468 (local address 127.0.192.195:38741)
0621 07:51:00.352702 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:00.352707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:00.352719 (+ 12us) server_negotiation.cc:407] Connection header received
0621 07:51:00.352893 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:00.352895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:00.352944 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:00.353044 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:51:00.353296 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:00.353770 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:00.354537 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:00.354735 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:00.354836 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:00.355112 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:51:00.355480 (+ 368us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:51:00.355623 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:00.355678 (+ 55us) server_negotiation.cc:299] Negotiation successful
0621 07:51:00.355723 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260621 07:51:00.361493 18614 raft_consensus.cc:1261] T f432687a62334f4e9264c145737c00d2 P c156aeefaa3045ef9987772ceb86b832 [term 1 FOLLOWER]: Refusing update from remote peer c19af2187aa34f0ba0eb85d263a05e78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:51:00.361531 18483 raft_consensus.cc:1261] T f432687a62334f4e9264c145737c00d2 P e823f8698d18455a8a37c4c7fa8ac561 [term 1 FOLLOWER]: Refusing update from remote peer c19af2187aa34f0ba0eb85d263a05e78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:51:00.362195 18818 consensus_queue.cc:1035] T f432687a62334f4e9264c145737c00d2 P c19af2187aa34f0ba0eb85d263a05e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c156aeefaa3045ef9987772ceb86b832" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 35855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:51:00.362372 18815 consensus_queue.cc:1035] T f432687a62334f4e9264c145737c00d2 P c19af2187aa34f0ba0eb85d263a05e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e823f8698d18455a8a37c4c7fa8ac561" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 42093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:51:00.376449 18341 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:51:00.376607 18341 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:51:00.378561 18341 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 81ed95aecbe043e0be6d54b169af9fff: Sending DeleteTablet for 3 replicas of tablet f432687a62334f4e9264c145737c00d2
I20260621 07:51:00.379181 18725 tablet_service.cc:1514] Processing DeleteTablet for tablet f432687a62334f4e9264c145737c00d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:51:00 UTC) from {username='slave'} at 127.0.0.1:40460
I20260621 07:51:00.379360 18594 tablet_service.cc:1514] Processing DeleteTablet for tablet f432687a62334f4e9264c145737c00d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:51:00 UTC) from {username='slave'} at 127.0.0.1:50600
I20260621 07:51:00.379462 18463 tablet_service.cc:1514] Processing DeleteTablet for tablet f432687a62334f4e9264c145737c00d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:51:00 UTC) from {username='slave'} at 127.0.0.1:60938
I20260621 07:51:00.379889 18833 tablet_replica.cc:330] T f432687a62334f4e9264c145737c00d2 P e823f8698d18455a8a37c4c7fa8ac561: stopping tablet replica
I20260621 07:51:00.379920 18832 tablet_replica.cc:330] T f432687a62334f4e9264c145737c00d2 P c156aeefaa3045ef9987772ceb86b832: stopping tablet replica
I20260621 07:51:00.380177 18832 raft_consensus.cc:2229] T f432687a62334f4e9264c145737c00d2 P c156aeefaa3045ef9987772ceb86b832 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:51:00.380223 18833 raft_consensus.cc:2229] T f432687a62334f4e9264c145737c00d2 P e823f8698d18455a8a37c4c7fa8ac561 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:51:00.380370 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 18401
I20260621 07:51:00.380432 18832 raft_consensus.cc:2258] T f432687a62334f4e9264c145737c00d2 P c156aeefaa3045ef9987772ceb86b832 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:51:00.381785 18832 ts_tablet_manager.cc:1905] T f432687a62334f4e9264c145737c00d2 P c156aeefaa3045ef9987772ceb86b832: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:51:00.384254 18831 tablet_replica.cc:330] T f432687a62334f4e9264c145737c00d2 P c19af2187aa34f0ba0eb85d263a05e78: stopping tablet replica
I20260621 07:51:00.384542 18831 raft_consensus.cc:2229] T f432687a62334f4e9264c145737c00d2 P c19af2187aa34f0ba0eb85d263a05e78 [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:51:00.384708 18832 ts_tablet_manager.cc:1918] T f432687a62334f4e9264c145737c00d2 P c156aeefaa3045ef9987772ceb86b832: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:51:00.384863 18832 log.cc:1196] T f432687a62334f4e9264c145737c00d2 P c156aeefaa3045ef9987772ceb86b832: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-1/wal/wals/f432687a62334f4e9264c145737c00d2
I20260621 07:51:00.384950 18831 raft_consensus.cc:2258] T f432687a62334f4e9264c145737c00d2 P c19af2187aa34f0ba0eb85d263a05e78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:51:00.385227 18832 ts_tablet_manager.cc:1939] T f432687a62334f4e9264c145737c00d2 P c156aeefaa3045ef9987772ceb86b832: Deleting consensus metadata
I20260621 07:51:00.386425 18326 catalog_manager.cc:4841] TS c156aeefaa3045ef9987772ceb86b832 (127.0.192.194:35855): tablet f432687a62334f4e9264c145737c00d2 (table test-table [id=899cb7cd27bc4ddc8c4ef1db433cb273]) successfully deleted
I20260621 07:51:00.386633 18831 ts_tablet_manager.cc:1905] T f432687a62334f4e9264c145737c00d2 P c19af2187aa34f0ba0eb85d263a05e78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:51:00.389294 18831 ts_tablet_manager.cc:1918] T f432687a62334f4e9264c145737c00d2 P c19af2187aa34f0ba0eb85d263a05e78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260621 07:51:00.389413 18326 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.192.193:42093 (error 108)
I20260621 07:51:00.389489 18831 log.cc:1196] T f432687a62334f4e9264c145737c00d2 P c19af2187aa34f0ba0eb85d263a05e78: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782028028522580-771-0/minicluster-data/ts-2/wal/wals/f432687a62334f4e9264c145737c00d2
I20260621 07:51:00.389673 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 18532
I20260621 07:51:00.390404 18831 ts_tablet_manager.cc:1939] T f432687a62334f4e9264c145737c00d2 P c19af2187aa34f0ba0eb85d263a05e78: Deleting consensus metadata
I20260621 07:51:00.391570 18327 catalog_manager.cc:4841] TS c19af2187aa34f0ba0eb85d263a05e78 (127.0.192.195:38741): tablet f432687a62334f4e9264c145737c00d2 (table test-table [id=899cb7cd27bc4ddc8c4ef1db433cb273]) successfully deleted
I20260621 07:51:00.392791 18806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:00.392467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:42093 (local address 127.0.0.1:60944)
0621 07:51:00.392625 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:00.392703 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.193:42093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260621 07:51:00.392941 18326 catalog_manager.cc:4584] TS e823f8698d18455a8a37c4c7fa8ac561 (127.0.192.193:42093): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f432687a62334f4e9264c145737c00d2: Network error: Client connection negotiation failed: client connection to 127.0.192.193:42093: connect: Connection refused (error 111)
I20260621 07:51:00.396909 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 18663
I20260621 07:51:00.401934 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 18310
2026-06-21T07:51:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1303 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:51:00 dist-test-slave-lzx3 krb5kdc[18837](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:51:00 dist-test-slave-lzx3 krb5kdc[18837](info): set up 2 sockets
Jun 21 07:51:00 dist-test-slave-lzx3 krb5kdc[18837](info): commencing operation
krb5kdc: starting...
W20260621 07:51:01.360440 771 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/krb5kdc/test-user.keytab.
Jun 21 07:51:01 dist-test-slave-lzx3 krb5kdc[18837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:51:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:51:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.192.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
I20260621 07:51:01.465613 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:44435
--webserver_interface=127.0.192.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:40221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:44435
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:01.564316 18850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:01.564592 18850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:01.564661 18850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:01.568384 18850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:51:01.568485 18850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:01.568527 18850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:01.568589 18850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:51:01.568647 18850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:51:01.573664 18850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:40221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:44435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:44435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:01.574582 18850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:01.575480 18850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:01.583415 18858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:01.583413 18855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:01.583468 18856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:01.583455 18850 server_base.cc:1034] running on GCE node
I20260621 07:51:01.584308 18850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:01.585248 18850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:01.586405 18850 hybrid_clock.cc:648] HybridClock initialized: now 1782028261586387 us; error 26 us; skew 500 ppm
Jun 21 07:51:01 dist-test-slave-lzx3 krb5kdc[18837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:01.589104 18850 init.cc:377] Logged in from keytab as kudu/127.0.192.254@KRBTEST.COM (short username kudu)
I20260621 07:51:01.589906 18850 webserver.cc:460] Webserver started at http://127.0.192.254:37063/ using document root <none> and password file <none>
I20260621 07:51:01.590376 18850 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:01.590476 18850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:01.590687 18850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:01.592523 18850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "2bb7aaabba5e4a0693ffb7f3923d2c78"
format_stamp: "Formatted at 2026-06-21 07:51:01 on dist-test-slave-lzx3"
server_key: "574bc0bb6bec774997666003dd5554fe"
server_key_iv: "c509c15705e124ba0302a0f3b7d48196"
server_key_version: "encryptionkey@0"
I20260621 07:51:01.593086 18850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "2bb7aaabba5e4a0693ffb7f3923d2c78"
format_stamp: "Formatted at 2026-06-21 07:51:01 on dist-test-slave-lzx3"
server_key: "574bc0bb6bec774997666003dd5554fe"
server_key_iv: "c509c15705e124ba0302a0f3b7d48196"
server_key_version: "encryptionkey@0"
I20260621 07:51:01.596295 18850 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:51:01.598512 18865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:01.599695 18850 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:51:01.599819 18850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "2bb7aaabba5e4a0693ffb7f3923d2c78"
format_stamp: "Formatted at 2026-06-21 07:51:01 on dist-test-slave-lzx3"
server_key: "574bc0bb6bec774997666003dd5554fe"
server_key_iv: "c509c15705e124ba0302a0f3b7d48196"
server_key_version: "encryptionkey@0"
I20260621 07:51:01.599915 18850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:01.614467 18850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:01.616722 18850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:01.616899 18850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:01.623240 18850 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:44435
I20260621 07:51:01.623250 18917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:44435 every 8 connection(s)
I20260621 07:51:01.624046 18850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:51:01.626312 18918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:01.630239 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 18850
I20260621 07:51:01.630333 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:51:01.630573 771 external_mini_cluster.cc:1428] Setting key 7d61ea9141c65d63bd4c4a29f77f7ed4
I20260621 07:51:01.631425 18918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bb7aaabba5e4a0693ffb7f3923d2c78: Bootstrap starting.
I20260621 07:51:01.633455 18918 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bb7aaabba5e4a0693ffb7f3923d2c78: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:01.634220 18918 log.cc:824] T 00000000000000000000000000000000 P 2bb7aaabba5e4a0693ffb7f3923d2c78: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:01.636008 18918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bb7aaabba5e4a0693ffb7f3923d2c78: No bootstrap required, opened a new log
Jun 21 07:51:01 dist-test-slave-lzx3 krb5kdc[18837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:51:01.639933 18918 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bb7aaabba5e4a0693ffb7f3923d2c78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bb7aaabba5e4a0693ffb7f3923d2c78" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 44435 } }
I20260621 07:51:01.640163 18918 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2bb7aaabba5e4a0693ffb7f3923d2c78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:01.640237 18918 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bb7aaabba5e4a0693ffb7f3923d2c78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bb7aaabba5e4a0693ffb7f3923d2c78, State: Initialized, Role: FOLLOWER
I20260621 07:51:01.640653 18918 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bb7aaabba5e4a0693ffb7f3923d2c78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bb7aaabba5e4a0693ffb7f3923d2c78" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 44435 } }
I20260621 07:51:01.640791 18918 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bb7aaabba5e4a0693ffb7f3923d2c78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:51:01.640878 18918 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bb7aaabba5e4a0693ffb7f3923d2c78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:51:01.640962 18918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bb7aaabba5e4a0693ffb7f3923d2c78 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:01.641711 18918 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bb7aaabba5e4a0693ffb7f3923d2c78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bb7aaabba5e4a0693ffb7f3923d2c78" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 44435 } }
I20260621 07:51:01.642006 18918 leader_election.cc:302] T 00000000000000000000000000000000 P 2bb7aaabba5e4a0693ffb7f3923d2c78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bb7aaabba5e4a0693ffb7f3923d2c78; no voters:
I20260621 07:51:01.642273 18918 leader_election.cc:288] T 00000000000000000000000000000000 P 2bb7aaabba5e4a0693ffb7f3923d2c78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:51:01.642369 18923 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bb7aaabba5e4a0693ffb7f3923d2c78 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:01.642570 18923 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bb7aaabba5e4a0693ffb7f3923d2c78 [term 1 LEADER]: Becoming Leader. State: Replica: 2bb7aaabba5e4a0693ffb7f3923d2c78, State: Running, Role: LEADER
I20260621 07:51:01.642913 18923 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bb7aaabba5e4a0693ffb7f3923d2c78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bb7aaabba5e4a0693ffb7f3923d2c78" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 44435 } }
I20260621 07:51:01.643393 18918 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bb7aaabba5e4a0693ffb7f3923d2c78 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:51:01.643989 18923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bb7aaabba5e4a0693ffb7f3923d2c78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bb7aaabba5e4a0693ffb7f3923d2c78. Latest consensus state: current_term: 1 leader_uuid: "2bb7aaabba5e4a0693ffb7f3923d2c78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bb7aaabba5e4a0693ffb7f3923d2c78" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 44435 } } }
I20260621 07:51:01.644167 18923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bb7aaabba5e4a0693ffb7f3923d2c78 [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:01.644366 18921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:01.631780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51106 (local address 127.0.192.254:44435)
0621 07:51:01.632148 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:01.632160 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:01.632188 (+ 28us) server_negotiation.cc:407] Connection header received
0621 07:51:01.632757 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:01.632772 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:01.633041 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:01.633264 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:01.634096 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:01.635162 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:01.636232 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:01.636755 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:01.636868 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:01.639423 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:01.639428 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:01.639448 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:01.639469 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:01.641238 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:01.641608 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:01.641613 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:01.641617 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:01.641717 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:01.641974 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:01.641977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:01.641979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:01.642461 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:01.643174 (+ 713us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:01.643298 (+ 124us) server_negotiation.cc:299] Negotiation successful
0621 07:51:01.643547 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":105,"threads_started":1}
I20260621 07:51:01.644378 18924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bb7aaabba5e4a0693ffb7f3923d2c78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2bb7aaabba5e4a0693ffb7f3923d2c78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bb7aaabba5e4a0693ffb7f3923d2c78" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 44435 } } }
I20260621 07:51:01.644502 18924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bb7aaabba5e4a0693ffb7f3923d2c78 [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:01.644744 18932 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:51:01.647109 18932 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:51:01.651647 18932 catalog_manager.cc:1347] Generated new cluster ID: 78115f3f46014a0e88a46bb48d27af61
I20260621 07:51:01.651698 18932 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:51:01.664810 18932 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:51:01.665491 18932 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:51:01.669340 18932 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2bb7aaabba5e4a0693ffb7f3923d2c78: Generated new TSK 0
I20260621 07:51:01.669808 18932 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.192.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab.
Entry for principal kudu/127.0.192.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab.
I20260621 07:51:01.697132 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.193:0
--local_ip_for_outbound_sockets=127.0.192.193
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:44435
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:40221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:01.792992 18944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:01.793174 18944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:01.793206 18944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:01.796466 18944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:01.796505 18944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:01.796586 18944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.193
I20260621 07:51:01.801057 18944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:40221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:44435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.193
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:01.801775 18944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:01.802583 18944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 07:51:01.810863 18944 server_base.cc:1034] running on GCE node
W20260621 07:51:01.810825 18952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:01.810829 18949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:01.810819 18950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:01.811403 18944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:01.811885 18944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:01.813050 18944 hybrid_clock.cc:648] HybridClock initialized: now 1782028261813027 us; error 36 us; skew 500 ppm
Jun 21 07:51:01 dist-test-slave-lzx3 krb5kdc[18837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:01.815337 18944 init.cc:377] Logged in from keytab as kudu/127.0.192.193@KRBTEST.COM (short username kudu)
I20260621 07:51:01.816043 18944 webserver.cc:460] Webserver started at http://127.0.192.193:35923/ using document root <none> and password file <none>
I20260621 07:51:01.816483 18944 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:01.816533 18944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:01.816766 18944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:01.818440 18944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-0/data/instance:
uuid: "fbacd5aced0f493590283cd8c82b29bc"
format_stamp: "Formatted at 2026-06-21 07:51:01 on dist-test-slave-lzx3"
server_key: "60ea05e15c0de55f62e1a7fea11f6463"
server_key_iv: "1849cb0e820a92dc824478e0e3fdc316"
server_key_version: "encryptionkey@0"
I20260621 07:51:01.818965 18944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-0/wal/instance:
uuid: "fbacd5aced0f493590283cd8c82b29bc"
format_stamp: "Formatted at 2026-06-21 07:51:01 on dist-test-slave-lzx3"
server_key: "60ea05e15c0de55f62e1a7fea11f6463"
server_key_iv: "1849cb0e820a92dc824478e0e3fdc316"
server_key_version: "encryptionkey@0"
I20260621 07:51:01.821990 18944 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 07:51:01.824069 18959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:01.825186 18944 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:51:01.825305 18944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-0/wal
uuid: "fbacd5aced0f493590283cd8c82b29bc"
format_stamp: "Formatted at 2026-06-21 07:51:01 on dist-test-slave-lzx3"
server_key: "60ea05e15c0de55f62e1a7fea11f6463"
server_key_iv: "1849cb0e820a92dc824478e0e3fdc316"
server_key_version: "encryptionkey@0"
I20260621 07:51:01.825399 18944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:01.846930 18944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:01.849375 18944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:01.849545 18944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:01.849990 18944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:51:01.850647 18944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:51:01.850721 18944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:01.850778 18944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:51:01.850807 18944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:01.858108 18944 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.193:46711
I20260621 07:51:01.858151 19072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.193:46711 every 8 connection(s)
I20260621 07:51:01.858860 18944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
I20260621 07:51:01.861799 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 18944
I20260621 07:51:01.861922 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-0/wal/instance
I20260621 07:51:01.862174 771 external_mini_cluster.cc:1428] Setting key 4ac02fcb7627cf7548cb8dd48b354e49
Jun 21 07:51:01 dist-test-slave-lzx3 krb5kdc[18837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.192.194@KRBTEST.COM; defaulting to no policy
I20260621 07:51:01.872074 18921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:01.860348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:51485 (local address 127.0.192.254:44435)
0621 07:51:01.860477 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:01.860480 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:01.861170 (+ 690us) server_negotiation.cc:407] Connection header received
0621 07:51:01.862465 (+ 1295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:01.862468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:01.862524 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:01.862632 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:01.863830 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:01.864321 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:01.865932 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:01.866139 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:01.866217 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:01.869001 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:01.869005 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:01.869007 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:01.869026 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:01.870284 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:01.870721 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:01.870724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:01.870725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:01.870789 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:01.871086 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:01.871088 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:01.871090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:01.871285 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:01.871377 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:01.871799 (+ 422us) server_negotiation.cc:299] Negotiation successful
0621 07:51:01.871933 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 07:51:01.872795 19075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:01.860648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:44435 (local address 127.0.192.193:51485)
0621 07:51:01.861022 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:01.861058 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:01.862250 (+ 1192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:01.862817 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:01.862825 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:01.863231 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:01.863668 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:01.863679 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:01.864452 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:01.864456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:01.865803 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:01.865817 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:01.866274 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:01.866281 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:01.866497 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:01.867014 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:01.867036 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:01.868830 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:01.870388 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:01.870393 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:01.870404 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:01.870618 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:01.870875 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:01.870877 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:01.870879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:01.870996 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:01.871375 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:01.871381 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:01.871582 (+ 201us) client_negotiation.cc:769] Sending connection context
0621 07:51:01.871748 (+ 166us) client_negotiation.cc:240] Negotiation successful
0621 07:51:01.872008 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":102,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.194@KRBTEST.COM" created.
I20260621 07:51:01.873797 19073 heartbeater.cc:347] Connected to a master server at 127.0.192.254:44435
I20260621 07:51:01.874018 19073 heartbeater.cc:464] Registering TS with master...
I20260621 07:51:01.874449 19073 heartbeater.cc:511] Master 127.0.192.254:44435 requested a full tablet report, sending...
I20260621 07:51:01.875815 18882 ts_manager.cc:194] Registered new tserver with Master: fbacd5aced0f493590283cd8c82b29bc (127.0.192.193:46711)
I20260621 07:51:01.876945 18882 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.193@KRBTEST.COM'} at 127.0.192.193:51485
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab.
Entry for principal kudu/127.0.192.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab.
I20260621 07:51:01.887036 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.194:0
--local_ip_for_outbound_sockets=127.0.192.194
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:44435
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:40221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:01.979759 19078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:01.980013 19078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:01.980077 19078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:01.983362 19078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:01.983433 19078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:01.983507 19078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.194
I20260621 07:51:01.987846 19078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:40221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:44435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.194
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:01.988690 19078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:01.989506 19078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:01.997241 19084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:01.997249 19086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:01.997272 19083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:01.997443 19078 server_base.cc:1034] running on GCE node
I20260621 07:51:01.997896 19078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:01.998373 19078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:01.999516 19078 hybrid_clock.cc:648] HybridClock initialized: now 1782028261999501 us; error 27 us; skew 500 ppm
Jun 21 07:51:01 dist-test-slave-lzx3 krb5kdc[18837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:02.001722 19078 init.cc:377] Logged in from keytab as kudu/127.0.192.194@KRBTEST.COM (short username kudu)
I20260621 07:51:02.002378 19078 webserver.cc:460] Webserver started at http://127.0.192.194:36965/ using document root <none> and password file <none>
I20260621 07:51:02.002851 19078 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:02.002902 19078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:02.003086 19078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:02.004716 19078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-1/data/instance:
uuid: "a69b959bfe9144f0b099bcb9265a2ba4"
format_stamp: "Formatted at 2026-06-21 07:51:01 on dist-test-slave-lzx3"
server_key: "d4f5e630c801e91125324b331f6fe356"
server_key_iv: "3ec16a4c76b1501e47c54e10c9499739"
server_key_version: "encryptionkey@0"
I20260621 07:51:02.005165 19078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-1/wal/instance:
uuid: "a69b959bfe9144f0b099bcb9265a2ba4"
format_stamp: "Formatted at 2026-06-21 07:51:01 on dist-test-slave-lzx3"
server_key: "d4f5e630c801e91125324b331f6fe356"
server_key_iv: "3ec16a4c76b1501e47c54e10c9499739"
server_key_version: "encryptionkey@0"
I20260621 07:51:02.008217 19078 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260621 07:51:02.010216 19093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:02.011160 19078 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 07:51:02.011258 19078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-1/wal
uuid: "a69b959bfe9144f0b099bcb9265a2ba4"
format_stamp: "Formatted at 2026-06-21 07:51:01 on dist-test-slave-lzx3"
server_key: "d4f5e630c801e91125324b331f6fe356"
server_key_iv: "3ec16a4c76b1501e47c54e10c9499739"
server_key_version: "encryptionkey@0"
I20260621 07:51:02.011353 19078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:02.034015 19078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:02.036146 19078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:02.036299 19078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:02.036799 19078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:51:02.037444 19078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:51:02.037487 19078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:02.037544 19078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:51:02.037597 19078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:02.045251 19078 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.194:37589
I20260621 07:51:02.045279 19206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.194:37589 every 8 connection(s)
I20260621 07:51:02.045938 19078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
I20260621 07:51:02.051992 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 19078
I20260621 07:51:02.052135 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-1/wal/instance
I20260621 07:51:02.052387 771 external_mini_cluster.cc:1428] Setting key fedfcc1ae22bc33b0f1861193545c97c
Jun 21 07:51:02 dist-test-slave-lzx3 krb5kdc[18837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:51:02.057577 18921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:02.047392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:52119 (local address 127.0.192.254:44435)
0621 07:51:02.047521 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:02.047525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:02.048233 (+ 708us) server_negotiation.cc:407] Connection header received
0621 07:51:02.049302 (+ 1069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:02.049309 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:02.049350 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:02.049415 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:02.050282 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:02.050725 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:02.051625 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:02.051874 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:02.051972 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:02.054337 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:02.054342 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:02.054348 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:02.054376 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:02.055683 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:02.056164 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:02.056167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:02.056169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:02.056248 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:02.056606 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:02.056608 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:02.056609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:02.056780 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:02.056850 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:02.057378 (+ 528us) server_negotiation.cc:299] Negotiation successful
0621 07:51:02.057480 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 07:51:02.058597 19209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:02.047701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:44435 (local address 127.0.192.194:52119)
0621 07:51:02.048092 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:02.048129 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:02.049141 (+ 1012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:02.049550 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:02.049555 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:02.049821 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:02.050177 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:02.050187 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:02.050816 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:02.050818 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:02.051534 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:02.051541 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:02.051994 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:02.051996 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:02.052153 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:02.052714 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:02.052732 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:02.054183 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:02.055794 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:02.055799 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:02.055816 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:02.056046 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:02.056336 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:02.056341 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:02.056345 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:02.056498 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:02.056875 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:02.056883 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:02.057129 (+ 246us) client_negotiation.cc:769] Sending connection context
0621 07:51:02.057300 (+ 171us) client_negotiation.cc:240] Negotiation successful
0621 07:51:02.057588 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":110,"threads_started":1}
I20260621 07:51:02.059628 19207 heartbeater.cc:347] Connected to a master server at 127.0.192.254:44435
I20260621 07:51:02.059819 19207 heartbeater.cc:464] Registering TS with master...
I20260621 07:51:02.060189 19207 heartbeater.cc:511] Master 127.0.192.254:44435 requested a full tablet report, sending...
I20260621 07:51:02.061070 18882 ts_manager.cc:194] Registered new tserver with Master: a69b959bfe9144f0b099bcb9265a2ba4 (127.0.192.194:37589)
WARNING: no policy specified for kudu/127.0.192.195@KRBTEST.COM; defaulting to no policy
I20260621 07:51:02.061600 18882 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.194@KRBTEST.COM'} at 127.0.192.194:52119
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab.
Entry for principal kudu/127.0.192.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab.
I20260621 07:51:02.077447 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.195:0
--local_ip_for_outbound_sockets=127.0.192.195
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:44435
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:40221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:02.171645 19212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:02.171836 19212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:02.171870 19212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:02.175374 19212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:02.175416 19212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:02.175462 19212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.195
I20260621 07:51:02.179934 19212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:40221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:44435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.195
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:02.180835 19212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:02.181582 19212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:02.189455 19218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:02.189432 19217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:02.189594 19212 server_base.cc:1034] running on GCE node
W20260621 07:51:02.189443 19220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:02.190066 19212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:02.190573 19212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:02.191751 19212 hybrid_clock.cc:648] HybridClock initialized: now 1782028262191726 us; error 44 us; skew 500 ppm
Jun 21 07:51:02 dist-test-slave-lzx3 krb5kdc[18837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028262, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:02.193928 19212 init.cc:377] Logged in from keytab as kudu/127.0.192.195@KRBTEST.COM (short username kudu)
I20260621 07:51:02.194603 19212 webserver.cc:460] Webserver started at http://127.0.192.195:36187/ using document root <none> and password file <none>
I20260621 07:51:02.195067 19212 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:02.195147 19212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:02.195341 19212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:02.197026 19212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-2/data/instance:
uuid: "666902a108734e2795a73752ed8f22e2"
format_stamp: "Formatted at 2026-06-21 07:51:02 on dist-test-slave-lzx3"
server_key: "2a21f15f6ead2f91ee4637a95ed2f3b0"
server_key_iv: "f59635e18959ce7abb190eea1d2926a4"
server_key_version: "encryptionkey@0"
I20260621 07:51:02.197497 19212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-2/wal/instance:
uuid: "666902a108734e2795a73752ed8f22e2"
format_stamp: "Formatted at 2026-06-21 07:51:02 on dist-test-slave-lzx3"
server_key: "2a21f15f6ead2f91ee4637a95ed2f3b0"
server_key_iv: "f59635e18959ce7abb190eea1d2926a4"
server_key_version: "encryptionkey@0"
I20260621 07:51:02.200618 19212 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:51:02.202617 19227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:02.203691 19212 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:51:02.203819 19212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-2/wal
uuid: "666902a108734e2795a73752ed8f22e2"
format_stamp: "Formatted at 2026-06-21 07:51:02 on dist-test-slave-lzx3"
server_key: "2a21f15f6ead2f91ee4637a95ed2f3b0"
server_key_iv: "f59635e18959ce7abb190eea1d2926a4"
server_key_version: "encryptionkey@0"
I20260621 07:51:02.203923 19212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:02.220798 19212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:02.223100 19212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:02.223218 19212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:02.223615 19212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:51:02.224227 19212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:51:02.224268 19212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:02.224299 19212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:51:02.224311 19212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:02.232280 19212 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.195:33849
I20260621 07:51:02.232300 19340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.195:33849 every 8 connection(s)
I20260621 07:51:02.233132 19212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
Jun 21 07:51:02 dist-test-slave-lzx3 krb5kdc[18837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028262, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:51:02.241901 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 19212
I20260621 07:51:02.242013 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-2/wal/instance
I20260621 07:51:02.242259 771 external_mini_cluster.cc:1428] Setting key 000bdb75448705bbc46c1d8374f8d99a
I20260621 07:51:02.245347 18921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:02.234668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:55893 (local address 127.0.192.254:44435)
0621 07:51:02.234792 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:02.234796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:02.235483 (+ 687us) server_negotiation.cc:407] Connection header received
0621 07:51:02.236586 (+ 1103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:02.236589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:02.236641 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:02.236732 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:02.237781 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:02.238673 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:02.239661 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:02.239864 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:02.239960 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:02.241984 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:02.241988 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:02.241990 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:02.242014 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:02.243496 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:02.243923 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:02.243927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:02.243929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:02.244005 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:02.244341 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:02.244345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:02.244347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:02.244516 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:02.244648 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:02.245084 (+ 436us) server_negotiation.cc:299] Negotiation successful
0621 07:51:02.245188 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 07:51:02.245990 19343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:02.234977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:44435 (local address 127.0.192.195:55893)
0621 07:51:02.235362 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:02.235396 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:02.236383 (+ 987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:02.236882 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:02.236887 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:02.237196 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:02.237635 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:02.237646 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:02.238801 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:02.238805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:02.239549 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:02.239557 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:02.239987 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:02.239990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:02.240137 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:02.240637 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:02.240650 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:02.241842 (+ 1192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:02.243597 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:02.243602 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:02.243613 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:02.243833 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:02.244103 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:02.244105 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:02.244107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:02.244245 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:02.244644 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:02.244649 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:02.244873 (+ 224us) client_negotiation.cc:769] Sending connection context
0621 07:51:02.245035 (+ 162us) client_negotiation.cc:240] Negotiation successful
0621 07:51:02.245297 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":105,"threads_started":1}
I20260621 07:51:02.246814 19341 heartbeater.cc:347] Connected to a master server at 127.0.192.254:44435
I20260621 07:51:02.247015 19341 heartbeater.cc:464] Registering TS with master...
I20260621 07:51:02.247382 19341 heartbeater.cc:511] Master 127.0.192.254:44435 requested a full tablet report, sending...
I20260621 07:51:02.248251 18882 ts_manager.cc:194] Registered new tserver with Master: 666902a108734e2795a73752ed8f22e2 (127.0.192.195:33849)
I20260621 07:51:02.248829 18882 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.195@KRBTEST.COM'} at 127.0.192.195:55893
I20260621 07:51:02.255779 771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:51:02.263712 18921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:02.257587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51120 (local address 127.0.192.254:44435)
0621 07:51:02.257750 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:02.257755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:02.257921 (+ 166us) server_negotiation.cc:407] Connection header received
0621 07:51:02.258008 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:02.258010 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:02.258044 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:02.258107 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:02.258725 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:02.259154 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:02.259877 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:02.260056 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:02.260117 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:02.261391 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:02.261393 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:02.261395 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:02.261413 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:02.262433 (+ 1020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:02.262829 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:02.262831 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:02.262831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:02.262893 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:02.263117 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:02.263119 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:02.263120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:02.263277 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:02.263376 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:02.263558 (+ 182us) server_negotiation.cc:299] Negotiation successful
0621 07:51:02.263622 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260621 07:51:02.266005 18882 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51120:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:51:02.268800 18882 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:51:02.280916 19356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:02.276116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59126 (local address 127.0.192.195:33849)
0621 07:51:02.276392 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:02.276400 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:02.276624 (+ 224us) server_negotiation.cc:407] Connection header received
0621 07:51:02.276873 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:02.276878 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:02.277063 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:02.277196 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:51:02.277420 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:02.278103 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:02.279699 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:02.280304 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:02.280414 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:02.280527 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:02.280716 (+ 189us) server_negotiation.cc:299] Negotiation successful
0621 07:51:02.280799 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":116,"threads_started":1}
I20260621 07:51:02.280907 19352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:02.275562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.195:33849 (local address 127.0.0.1:59126)
0621 07:51:02.276030 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:02.276546 (+ 516us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:02.276642 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:02.277238 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:02.277243 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:02.277263 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:51:02.277331 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:02.277335 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:02.278239 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:02.278243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:02.279575 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:02.279585 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:02.280414 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:02.280417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:02.280506 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:02.280534 (+ 28us) client_negotiation.cc:769] Sending connection context
0621 07:51:02.280688 (+ 154us) client_negotiation.cc:240] Negotiation successful
0621 07:51:02.280793 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":47,"thread_start_us":75,"threads_started":1}
I20260621 07:51:02.281298 19354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:02.275602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48594 (local address 127.0.192.194:37589)
0621 07:51:02.276421 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:02.276432 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:02.276454 (+ 22us) server_negotiation.cc:407] Connection header received
0621 07:51:02.276521 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:02.276526 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:02.276747 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:02.276876 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:51:02.277318 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:02.278067 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:02.279603 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:02.280306 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:02.280400 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:02.280509 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:02.280967 (+ 458us) server_negotiation.cc:299] Negotiation successful
0621 07:51:02.281179 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":733,"thread_start_us":114,"threads_started":1}
I20260621 07:51:02.280915 19353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:02.275488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:37589 (local address 127.0.0.1:48594)
0621 07:51:02.276030 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:02.276075 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:02.276176 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:02.277064 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:02.277068 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:02.277087 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:51:02.277201 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:02.277209 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:02.278207 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:02.278211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:02.279370 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:02.279378 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:02.280580 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:02.280584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:02.280660 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:02.280677 (+ 17us) client_negotiation.cc:769] Sending connection context
0621 07:51:02.280743 (+ 66us) client_negotiation.cc:240] Negotiation successful
0621 07:51:02.280800 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":33,"thread_start_us":54,"threads_started":1}
I20260621 07:51:02.282285 19357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:02.276157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49140 (local address 127.0.192.193:46711)
0621 07:51:02.276860 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:02.276870 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:02.278417 (+ 1547us) server_negotiation.cc:407] Connection header received
0621 07:51:02.278520 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:02.278525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:02.278710 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:02.278873 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:51:02.279045 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:02.279922 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:02.281080 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:02.281788 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:02.281896 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:02.281996 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:02.282102 (+ 106us) server_negotiation.cc:299] Negotiation successful
0621 07:51:02.282183 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":139,"threads_started":1}
I20260621 07:51:02.283645 19275 tablet_service.cc:1467] Processing CreateTablet for tablet a1b9748f6b9f4d40a1530f47eb222161 (DEFAULT_TABLE table=test-table [id=02e3f2b80cb848d99cf2b0685f6768b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:51:02.283648 19141 tablet_service.cc:1467] Processing CreateTablet for tablet a1b9748f6b9f4d40a1530f47eb222161 (DEFAULT_TABLE table=test-table [id=02e3f2b80cb848d99cf2b0685f6768b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:51:02.284627 19275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1b9748f6b9f4d40a1530f47eb222161. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:02.284626 19141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1b9748f6b9f4d40a1530f47eb222161. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:02.288775 19355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:02.275817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:46711 (local address 127.0.0.1:49140)
0621 07:51:02.278312 (+ 2495us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:02.278324 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:02.278416 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:02.278876 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:02.278878 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:02.278894 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:51:02.278963 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:02.278967 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:02.280035 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:02.280038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:02.280970 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:02.280980 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:02.281903 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:02.281906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:02.281982 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:02.282002 (+ 20us) client_negotiation.cc:769] Sending connection context
0621 07:51:02.282075 (+ 73us) client_negotiation.cc:240] Negotiation successful
0621 07:51:02.288684 (+ 6609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2437,"thread_start_us":79,"threads_started":1}
I20260621 07:51:02.291111 19358 tablet_bootstrap.cc:492] T a1b9748f6b9f4d40a1530f47eb222161 P 666902a108734e2795a73752ed8f22e2: Bootstrap starting.
I20260621 07:51:02.290894 19007 tablet_service.cc:1467] Processing CreateTablet for tablet a1b9748f6b9f4d40a1530f47eb222161 (DEFAULT_TABLE table=test-table [id=02e3f2b80cb848d99cf2b0685f6768b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:51:02.291755 19007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1b9748f6b9f4d40a1530f47eb222161. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:02.292816 19359 tablet_bootstrap.cc:492] T a1b9748f6b9f4d40a1530f47eb222161 P a69b959bfe9144f0b099bcb9265a2ba4: Bootstrap starting.
I20260621 07:51:02.293006 19358 tablet_bootstrap.cc:654] T a1b9748f6b9f4d40a1530f47eb222161 P 666902a108734e2795a73752ed8f22e2: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:02.294060 19358 log.cc:824] T a1b9748f6b9f4d40a1530f47eb222161 P 666902a108734e2795a73752ed8f22e2: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:02.294507 19359 tablet_bootstrap.cc:654] T a1b9748f6b9f4d40a1530f47eb222161 P a69b959bfe9144f0b099bcb9265a2ba4: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:02.295101 19359 log.cc:824] T a1b9748f6b9f4d40a1530f47eb222161 P a69b959bfe9144f0b099bcb9265a2ba4: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:02.295799 19358 tablet_bootstrap.cc:492] T a1b9748f6b9f4d40a1530f47eb222161 P 666902a108734e2795a73752ed8f22e2: No bootstrap required, opened a new log
I20260621 07:51:02.295946 19358 ts_tablet_manager.cc:1397] T a1b9748f6b9f4d40a1530f47eb222161 P 666902a108734e2795a73752ed8f22e2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260621 07:51:02.296386 19359 tablet_bootstrap.cc:492] T a1b9748f6b9f4d40a1530f47eb222161 P a69b959bfe9144f0b099bcb9265a2ba4: No bootstrap required, opened a new log
I20260621 07:51:02.296499 19359 ts_tablet_manager.cc:1397] T a1b9748f6b9f4d40a1530f47eb222161 P a69b959bfe9144f0b099bcb9265a2ba4: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260621 07:51:02.298125 19362 tablet_bootstrap.cc:492] T a1b9748f6b9f4d40a1530f47eb222161 P fbacd5aced0f493590283cd8c82b29bc: Bootstrap starting.
I20260621 07:51:02.299238 19358 raft_consensus.cc:348] T a1b9748f6b9f4d40a1530f47eb222161 P 666902a108734e2795a73752ed8f22e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a69b959bfe9144f0b099bcb9265a2ba4" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 37589 } } peers { permanent_uuid: "666902a108734e2795a73752ed8f22e2" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 33849 } } peers { permanent_uuid: "fbacd5aced0f493590283cd8c82b29bc" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46711 } }
I20260621 07:51:02.299438 19358 raft_consensus.cc:374] T a1b9748f6b9f4d40a1530f47eb222161 P 666902a108734e2795a73752ed8f22e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:02.299482 19358 raft_consensus.cc:729] T a1b9748f6b9f4d40a1530f47eb222161 P 666902a108734e2795a73752ed8f22e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 666902a108734e2795a73752ed8f22e2, State: Initialized, Role: FOLLOWER
I20260621 07:51:02.299695 19359 raft_consensus.cc:348] T a1b9748f6b9f4d40a1530f47eb222161 P a69b959bfe9144f0b099bcb9265a2ba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a69b959bfe9144f0b099bcb9265a2ba4" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 37589 } } peers { permanent_uuid: "666902a108734e2795a73752ed8f22e2" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 33849 } } peers { permanent_uuid: "fbacd5aced0f493590283cd8c82b29bc" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46711 } }
I20260621 07:51:02.299855 19358 consensus_queue.cc:260] T a1b9748f6b9f4d40a1530f47eb222161 P 666902a108734e2795a73752ed8f22e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a69b959bfe9144f0b099bcb9265a2ba4" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 37589 } } peers { permanent_uuid: "666902a108734e2795a73752ed8f22e2" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 33849 } } peers { permanent_uuid: "fbacd5aced0f493590283cd8c82b29bc" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46711 } }
I20260621 07:51:02.299926 19359 raft_consensus.cc:374] T a1b9748f6b9f4d40a1530f47eb222161 P a69b959bfe9144f0b099bcb9265a2ba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:02.299978 19359 raft_consensus.cc:729] T a1b9748f6b9f4d40a1530f47eb222161 P a69b959bfe9144f0b099bcb9265a2ba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a69b959bfe9144f0b099bcb9265a2ba4, State: Initialized, Role: FOLLOWER
I20260621 07:51:02.300339 19362 tablet_bootstrap.cc:654] T a1b9748f6b9f4d40a1530f47eb222161 P fbacd5aced0f493590283cd8c82b29bc: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:02.300381 19341 heartbeater.cc:503] Master 127.0.192.254:44435 was elected leader, sending a full tablet report...
I20260621 07:51:02.300411 19359 consensus_queue.cc:260] T a1b9748f6b9f4d40a1530f47eb222161 P a69b959bfe9144f0b099bcb9265a2ba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a69b959bfe9144f0b099bcb9265a2ba4" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 37589 } } peers { permanent_uuid: "666902a108734e2795a73752ed8f22e2" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 33849 } } peers { permanent_uuid: "fbacd5aced0f493590283cd8c82b29bc" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46711 } }
I20260621 07:51:02.300709 19358 ts_tablet_manager.cc:1428] T a1b9748f6b9f4d40a1530f47eb222161 P 666902a108734e2795a73752ed8f22e2: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260621 07:51:02.301046 19362 log.cc:824] T a1b9748f6b9f4d40a1530f47eb222161 P fbacd5aced0f493590283cd8c82b29bc: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:02.301220 19359 ts_tablet_manager.cc:1428] T a1b9748f6b9f4d40a1530f47eb222161 P a69b959bfe9144f0b099bcb9265a2ba4: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260621 07:51:02.301559 19207 heartbeater.cc:503] Master 127.0.192.254:44435 was elected leader, sending a full tablet report...
I20260621 07:51:02.302736 19362 tablet_bootstrap.cc:492] T a1b9748f6b9f4d40a1530f47eb222161 P fbacd5aced0f493590283cd8c82b29bc: No bootstrap required, opened a new log
I20260621 07:51:02.302908 19362 ts_tablet_manager.cc:1397] T a1b9748f6b9f4d40a1530f47eb222161 P fbacd5aced0f493590283cd8c82b29bc: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260621 07:51:02.305629 19362 raft_consensus.cc:348] T a1b9748f6b9f4d40a1530f47eb222161 P fbacd5aced0f493590283cd8c82b29bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a69b959bfe9144f0b099bcb9265a2ba4" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 37589 } } peers { permanent_uuid: "666902a108734e2795a73752ed8f22e2" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 33849 } } peers { permanent_uuid: "fbacd5aced0f493590283cd8c82b29bc" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46711 } }
I20260621 07:51:02.305827 19362 raft_consensus.cc:374] T a1b9748f6b9f4d40a1530f47eb222161 P fbacd5aced0f493590283cd8c82b29bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:02.305879 19362 raft_consensus.cc:729] T a1b9748f6b9f4d40a1530f47eb222161 P fbacd5aced0f493590283cd8c82b29bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbacd5aced0f493590283cd8c82b29bc, State: Initialized, Role: FOLLOWER
I20260621 07:51:02.306177 19362 consensus_queue.cc:260] T a1b9748f6b9f4d40a1530f47eb222161 P fbacd5aced0f493590283cd8c82b29bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a69b959bfe9144f0b099bcb9265a2ba4" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 37589 } } peers { permanent_uuid: "666902a108734e2795a73752ed8f22e2" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 33849 } } peers { permanent_uuid: "fbacd5aced0f493590283cd8c82b29bc" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46711 } }
I20260621 07:51:02.306692 19073 heartbeater.cc:503] Master 127.0.192.254:44435 was elected leader, sending a full tablet report...
I20260621 07:51:02.307006 19362 ts_tablet_manager.cc:1428] T a1b9748f6b9f4d40a1530f47eb222161 P fbacd5aced0f493590283cd8c82b29bc: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260621 07:51:02.359807 19074 tablet.cc:2369] T a1b9748f6b9f4d40a1530f47eb222161 P fbacd5aced0f493590283cd8c82b29bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:51:02.402802 19366 raft_consensus.cc:482] T a1b9748f6b9f4d40a1530f47eb222161 P fbacd5aced0f493590283cd8c82b29bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:51:02.402980 19366 raft_consensus.cc:504] T a1b9748f6b9f4d40a1530f47eb222161 P fbacd5aced0f493590283cd8c82b29bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a69b959bfe9144f0b099bcb9265a2ba4" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 37589 } } peers { permanent_uuid: "666902a108734e2795a73752ed8f22e2" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 33849 } } peers { permanent_uuid: "fbacd5aced0f493590283cd8c82b29bc" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46711 } }
I20260621 07:51:02.403873 19366 leader_election.cc:288] T a1b9748f6b9f4d40a1530f47eb222161 P fbacd5aced0f493590283cd8c82b29bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a69b959bfe9144f0b099bcb9265a2ba4 (127.0.192.194:37589), 666902a108734e2795a73752ed8f22e2 (127.0.192.195:33849)
I20260621 07:51:02.407553 19368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:02.404100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.195:33849 (local address 127.0.192.193:41689)
0621 07:51:02.404366 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:02.404379 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:02.404483 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:02.404809 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:02.404814 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:02.404833 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:51:02.404946 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:02.404951 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:02.405657 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:02.405660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:02.406405 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:02.406413 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:02.407089 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:02.407102 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:02.407199 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:02.407234 (+ 35us) client_negotiation.cc:769] Sending connection context
0621 07:51:02.407305 (+ 71us) client_negotiation.cc:240] Negotiation successful
0621 07:51:02.407371 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":58,"thread_start_us":79,"threads_started":1}
I20260621 07:51:02.407567 19356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:02.404202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:41689 (local address 127.0.192.195:33849)
0621 07:51:02.404322 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:02.404326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:02.404462 (+ 136us) server_negotiation.cc:407] Connection header received
0621 07:51:02.404626 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:02.404638 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:02.404692 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:02.404759 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:51:02.405079 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:02.405544 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:02.406506 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:02.406980 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:02.407069 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:02.407105 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:02.407326 (+ 221us) server_negotiation.cc:299] Negotiation successful
0621 07:51:02.407394 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260621 07:51:02.407770 19354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:02.404215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:49499 (local address 127.0.192.194:37589)
0621 07:51:02.404388 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:02.404394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:02.404538 (+ 144us) server_negotiation.cc:407] Connection header received
0621 07:51:02.404696 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:02.404698 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:02.404784 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:02.404866 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:51:02.405165 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:02.405560 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:02.406566 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:02.407188 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:02.407314 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:02.407356 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:02.407557 (+ 201us) server_negotiation.cc:299] Negotiation successful
0621 07:51:02.407654 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260621 07:51:02.407744 19367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:02.404101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:37589 (local address 127.0.192.193:49499)
0621 07:51:02.404419 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:02.404449 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:02.404584 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:02.404893 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:02.404895 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:02.404907 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:51:02.405048 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:02.405053 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:02.405687 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:02.405689 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:02.406435 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:02.406443 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:02.407321 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:02.407325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:02.407423 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:02.407444 (+ 21us) client_negotiation.cc:769] Sending connection context
0621 07:51:02.407538 (+ 94us) client_negotiation.cc:240] Negotiation successful
0621 07:51:02.407604 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":30,"thread_start_us":57,"threads_started":1}
I20260621 07:51:02.408412 19161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1b9748f6b9f4d40a1530f47eb222161" candidate_uuid: "fbacd5aced0f493590283cd8c82b29bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a69b959bfe9144f0b099bcb9265a2ba4" is_pre_election: true
I20260621 07:51:02.408550 19295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1b9748f6b9f4d40a1530f47eb222161" candidate_uuid: "fbacd5aced0f493590283cd8c82b29bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "666902a108734e2795a73752ed8f22e2" is_pre_election: true
I20260621 07:51:02.408712 19161 raft_consensus.cc:2454] T a1b9748f6b9f4d40a1530f47eb222161 P a69b959bfe9144f0b099bcb9265a2ba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbacd5aced0f493590283cd8c82b29bc in term 0.
I20260621 07:51:02.408850 19295 raft_consensus.cc:2454] T a1b9748f6b9f4d40a1530f47eb222161 P 666902a108734e2795a73752ed8f22e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbacd5aced0f493590283cd8c82b29bc in term 0.
I20260621 07:51:02.409148 18960 leader_election.cc:302] T a1b9748f6b9f4d40a1530f47eb222161 P fbacd5aced0f493590283cd8c82b29bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a69b959bfe9144f0b099bcb9265a2ba4, fbacd5aced0f493590283cd8c82b29bc; no voters:
I20260621 07:51:02.409368 19366 raft_consensus.cc:2781] T a1b9748f6b9f4d40a1530f47eb222161 P fbacd5aced0f493590283cd8c82b29bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:51:02.409449 19366 raft_consensus.cc:482] T a1b9748f6b9f4d40a1530f47eb222161 P fbacd5aced0f493590283cd8c82b29bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:51:02.409502 19366 raft_consensus.cc:3037] T a1b9748f6b9f4d40a1530f47eb222161 P fbacd5aced0f493590283cd8c82b29bc [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:02.410348 19366 raft_consensus.cc:504] T a1b9748f6b9f4d40a1530f47eb222161 P fbacd5aced0f493590283cd8c82b29bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a69b959bfe9144f0b099bcb9265a2ba4" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 37589 } } peers { permanent_uuid: "666902a108734e2795a73752ed8f22e2" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 33849 } } peers { permanent_uuid: "fbacd5aced0f493590283cd8c82b29bc" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46711 } }
I20260621 07:51:02.410669 19366 leader_election.cc:288] T a1b9748f6b9f4d40a1530f47eb222161 P fbacd5aced0f493590283cd8c82b29bc [CANDIDATE]: Term 1 election: Requested vote from peers a69b959bfe9144f0b099bcb9265a2ba4 (127.0.192.194:37589), 666902a108734e2795a73752ed8f22e2 (127.0.192.195:33849)
I20260621 07:51:02.410995 19295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1b9748f6b9f4d40a1530f47eb222161" candidate_uuid: "fbacd5aced0f493590283cd8c82b29bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "666902a108734e2795a73752ed8f22e2"
I20260621 07:51:02.410995 19161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1b9748f6b9f4d40a1530f47eb222161" candidate_uuid: "fbacd5aced0f493590283cd8c82b29bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a69b959bfe9144f0b099bcb9265a2ba4"
I20260621 07:51:02.411098 19295 raft_consensus.cc:3037] T a1b9748f6b9f4d40a1530f47eb222161 P 666902a108734e2795a73752ed8f22e2 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:02.411113 19161 raft_consensus.cc:3037] T a1b9748f6b9f4d40a1530f47eb222161 P a69b959bfe9144f0b099bcb9265a2ba4 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:02.412202 19161 raft_consensus.cc:2454] T a1b9748f6b9f4d40a1530f47eb222161 P a69b959bfe9144f0b099bcb9265a2ba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbacd5aced0f493590283cd8c82b29bc in term 1.
I20260621 07:51:02.412202 19295 raft_consensus.cc:2454] T a1b9748f6b9f4d40a1530f47eb222161 P 666902a108734e2795a73752ed8f22e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbacd5aced0f493590283cd8c82b29bc in term 1.
I20260621 07:51:02.412446 18960 leader_election.cc:302] T a1b9748f6b9f4d40a1530f47eb222161 P fbacd5aced0f493590283cd8c82b29bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a69b959bfe9144f0b099bcb9265a2ba4, fbacd5aced0f493590283cd8c82b29bc; no voters:
I20260621 07:51:02.412647 19366 raft_consensus.cc:2781] T a1b9748f6b9f4d40a1530f47eb222161 P fbacd5aced0f493590283cd8c82b29bc [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:02.412894 19366 raft_consensus.cc:686] T a1b9748f6b9f4d40a1530f47eb222161 P fbacd5aced0f493590283cd8c82b29bc [term 1 LEADER]: Becoming Leader. State: Replica: fbacd5aced0f493590283cd8c82b29bc, State: Running, Role: LEADER
I20260621 07:51:02.413167 19366 consensus_queue.cc:237] T a1b9748f6b9f4d40a1530f47eb222161 P fbacd5aced0f493590283cd8c82b29bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a69b959bfe9144f0b099bcb9265a2ba4" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 37589 } } peers { permanent_uuid: "666902a108734e2795a73752ed8f22e2" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 33849 } } peers { permanent_uuid: "fbacd5aced0f493590283cd8c82b29bc" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46711 } }
I20260621 07:51:02.415879 18880 catalog_manager.cc:5495] T a1b9748f6b9f4d40a1530f47eb222161 P fbacd5aced0f493590283cd8c82b29bc reported cstate change: term changed from 0 to 1, leader changed from <none> to fbacd5aced0f493590283cd8c82b29bc (127.0.192.193). New cstate: current_term: 1 leader_uuid: "fbacd5aced0f493590283cd8c82b29bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a69b959bfe9144f0b099bcb9265a2ba4" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 37589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "666902a108734e2795a73752ed8f22e2" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 33849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbacd5aced0f493590283cd8c82b29bc" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 46711 } health_report { overall_health: HEALTHY } } }
I20260621 07:51:02.436069 19357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:02.432846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49156 (local address 127.0.192.193:46711)
0621 07:51:02.432989 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:02.433005 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:02.433016 (+ 11us) server_negotiation.cc:407] Connection header received
0621 07:51:02.433175 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:02.433178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:02.433233 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:02.433349 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:51:02.433579 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:02.433992 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:02.434717 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:02.434918 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:02.435012 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:02.435355 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:51:02.435688 (+ 333us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:51:02.435827 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:02.435921 (+ 94us) server_negotiation.cc:299] Negotiation successful
0621 07:51:02.435953 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 07:51:02.440936 19161 raft_consensus.cc:1261] T a1b9748f6b9f4d40a1530f47eb222161 P a69b959bfe9144f0b099bcb9265a2ba4 [term 1 FOLLOWER]: Refusing update from remote peer fbacd5aced0f493590283cd8c82b29bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:51:02.440958 19295 raft_consensus.cc:1261] T a1b9748f6b9f4d40a1530f47eb222161 P 666902a108734e2795a73752ed8f22e2 [term 1 FOLLOWER]: Refusing update from remote peer fbacd5aced0f493590283cd8c82b29bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:51:02.441514 19369 consensus_queue.cc:1035] T a1b9748f6b9f4d40a1530f47eb222161 P fbacd5aced0f493590283cd8c82b29bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "666902a108734e2795a73752ed8f22e2" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 33849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:51:02.441672 19366 consensus_queue.cc:1035] T a1b9748f6b9f4d40a1530f47eb222161 P fbacd5aced0f493590283cd8c82b29bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a69b959bfe9144f0b099bcb9265a2ba4" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 37589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:51:02.448216 19373 mvcc.cc:205] Tried to move back new op lower bound from 7299187762952380416 to 7299187762846908416. Current Snapshot: MvccSnapshot[applied={T|T < 7299187762952380416}]
I20260621 07:51:02.449234 19374 mvcc.cc:205] Tried to move back new op lower bound from 7299187762952380416 to 7299187762846908416. Current Snapshot: MvccSnapshot[applied={T|T < 7299187762952380416}]
I20260621 07:51:02.455521 18880 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:51:02.455660 18880 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:51:02.458088 18880 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2bb7aaabba5e4a0693ffb7f3923d2c78: Sending DeleteTablet for 3 replicas of tablet a1b9748f6b9f4d40a1530f47eb222161
I20260621 07:51:02.458818 19141 tablet_service.cc:1514] Processing DeleteTablet for tablet a1b9748f6b9f4d40a1530f47eb222161 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:51:02 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:48594
I20260621 07:51:02.458833 19275 tablet_service.cc:1514] Processing DeleteTablet for tablet a1b9748f6b9f4d40a1530f47eb222161 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:51:02 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:59126
I20260621 07:51:02.459260 19382 tablet_replica.cc:330] T a1b9748f6b9f4d40a1530f47eb222161 P a69b959bfe9144f0b099bcb9265a2ba4: stopping tablet replica
I20260621 07:51:02.459456 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 18944
I20260621 07:51:02.459487 19007 tablet_service.cc:1514] Processing DeleteTablet for tablet a1b9748f6b9f4d40a1530f47eb222161 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:51:02 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:49140
I20260621 07:51:02.459699 19382 raft_consensus.cc:2229] T a1b9748f6b9f4d40a1530f47eb222161 P a69b959bfe9144f0b099bcb9265a2ba4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:51:02.460022 19382 raft_consensus.cc:2258] T a1b9748f6b9f4d40a1530f47eb222161 P a69b959bfe9144f0b099bcb9265a2ba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:51:02.460363 19383 tablet_replica.cc:330] T a1b9748f6b9f4d40a1530f47eb222161 P 666902a108734e2795a73752ed8f22e2: stopping tablet replica
I20260621 07:51:02.460624 19383 raft_consensus.cc:2229] T a1b9748f6b9f4d40a1530f47eb222161 P 666902a108734e2795a73752ed8f22e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:51:02.460879 19383 raft_consensus.cc:2258] T a1b9748f6b9f4d40a1530f47eb222161 P 666902a108734e2795a73752ed8f22e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:51:02.461535 19382 ts_tablet_manager.cc:1905] T a1b9748f6b9f4d40a1530f47eb222161 P a69b959bfe9144f0b099bcb9265a2ba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:51:02.462874 19383 ts_tablet_manager.cc:1905] T a1b9748f6b9f4d40a1530f47eb222161 P 666902a108734e2795a73752ed8f22e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:51:02.463974 19382 ts_tablet_manager.cc:1918] T a1b9748f6b9f4d40a1530f47eb222161 P a69b959bfe9144f0b099bcb9265a2ba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:51:02.464073 19382 log.cc:1196] T a1b9748f6b9f4d40a1530f47eb222161 P a69b959bfe9144f0b099bcb9265a2ba4: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-1/wal/wals/a1b9748f6b9f4d40a1530f47eb222161
I20260621 07:51:02.464427 19382 ts_tablet_manager.cc:1939] T a1b9748f6b9f4d40a1530f47eb222161 P a69b959bfe9144f0b099bcb9265a2ba4: Deleting consensus metadata
I20260621 07:51:02.465267 19383 ts_tablet_manager.cc:1918] T a1b9748f6b9f4d40a1530f47eb222161 P 666902a108734e2795a73752ed8f22e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:51:02.465348 19383 log.cc:1196] T a1b9748f6b9f4d40a1530f47eb222161 P 666902a108734e2795a73752ed8f22e2: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782028028522580-771-0/minicluster-data/ts-2/wal/wals/a1b9748f6b9f4d40a1530f47eb222161
I20260621 07:51:02.465592 18866 catalog_manager.cc:4841] TS a69b959bfe9144f0b099bcb9265a2ba4 (127.0.192.194:37589): tablet a1b9748f6b9f4d40a1530f47eb222161 (table test-table [id=02e3f2b80cb848d99cf2b0685f6768b3]) successfully deleted
I20260621 07:51:02.465653 19383 ts_tablet_manager.cc:1939] T a1b9748f6b9f4d40a1530f47eb222161 P 666902a108734e2795a73752ed8f22e2: Deleting consensus metadata
W20260621 07:51:02.466437 18869 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.192.193:46711 (error 108)
I20260621 07:51:02.466796 18868 catalog_manager.cc:4841] TS 666902a108734e2795a73752ed8f22e2 (127.0.192.195:33849): tablet a1b9748f6b9f4d40a1530f47eb222161 (table test-table [id=02e3f2b80cb848d99cf2b0685f6768b3]) successfully deleted
I20260621 07:51:02.467223 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 19078
I20260621 07:51:02.467473 19355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:02.467209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:46711 (local address 127.0.0.1:49170)
0621 07:51:02.467320 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:02.467388 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.193:46711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260621 07:51:02.467808 18869 catalog_manager.cc:4584] TS fbacd5aced0f493590283cd8c82b29bc (127.0.192.193:46711): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a1b9748f6b9f4d40a1530f47eb222161: Network error: Client connection negotiation failed: client connection to 127.0.192.193:46711: connect: Connection refused (error 111)
I20260621 07:51:02.472748 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 19212
I20260621 07:51:02.477484 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 18850
2026-06-21T07:51:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2075 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-21T07:51:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:51:02Z Disabled control of system clock
I20260621 07:51:02.503865 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:39075
--webserver_interface=127.0.192.254
--webserver_port=0
--builtin_ntp_servers=127.0.192.212:36153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:39075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 07:51:02.600518 19388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:02.600720 19388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:02.600749 19388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:02.604364 19388 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:51:02.604409 19388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:02.604423 19388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:02.604437 19388 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:51:02.604451 19388 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:51:02.608783 19388 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:36153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:39075
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:39075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:02.609552 19388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:02.610618 19388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:02.617910 19394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:02.617914 19396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:02.617959 19388 server_base.cc:1034] running on GCE node
W20260621 07:51:02.617939 19393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:02.618671 19388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:02.619577 19388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:02.620783 19388 hybrid_clock.cc:648] HybridClock initialized: now 1782028262620759 us; error 29 us; skew 500 ppm
I20260621 07:51:02.622242 19388 webserver.cc:460] Webserver started at http://127.0.192.254:44367/ using document root <none> and password file <none>
I20260621 07:51:02.622740 19388 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:02.622845 19388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:02.623039 19388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:02.624684 19388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "5a122f049a6e4791982a202390cb924f"
format_stamp: "Formatted at 2026-06-21 07:51:02 on dist-test-slave-lzx3"
server_key: "00d800e57596fcce4bcdaef13b74c924"
server_key_iv: "5e9c79e6956c2a1139c93fd2acf246f8"
server_key_version: "encryptionkey@0"
I20260621 07:51:02.625186 19388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "5a122f049a6e4791982a202390cb924f"
format_stamp: "Formatted at 2026-06-21 07:51:02 on dist-test-slave-lzx3"
server_key: "00d800e57596fcce4bcdaef13b74c924"
server_key_iv: "5e9c79e6956c2a1139c93fd2acf246f8"
server_key_version: "encryptionkey@0"
I20260621 07:51:02.628328 19388 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:51:02.630333 19402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:02.631278 19388 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:51:02.631393 19388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "5a122f049a6e4791982a202390cb924f"
format_stamp: "Formatted at 2026-06-21 07:51:02 on dist-test-slave-lzx3"
server_key: "00d800e57596fcce4bcdaef13b74c924"
server_key_iv: "5e9c79e6956c2a1139c93fd2acf246f8"
server_key_version: "encryptionkey@0"
I20260621 07:51:02.631482 19388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:02.641708 19388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:02.642221 19388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:02.642385 19388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:02.648280 19388 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:39075
I20260621 07:51:02.648305 19454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:39075 every 8 connection(s)
I20260621 07:51:02.649085 19388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:51:02.651674 19455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:02.656319 19455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a122f049a6e4791982a202390cb924f: Bootstrap starting.
I20260621 07:51:02.658187 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 19388
I20260621 07:51:02.658324 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:51:02.658573 19455 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a122f049a6e4791982a202390cb924f: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:02.658594 771 external_mini_cluster.cc:1428] Setting key 2af22acf5fbcd6e461e784db115ee30e
I20260621 07:51:02.659382 19455 log.cc:824] T 00000000000000000000000000000000 P 5a122f049a6e4791982a202390cb924f: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:02.661161 19455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a122f049a6e4791982a202390cb924f: No bootstrap required, opened a new log
I20260621 07:51:02.663926 19455 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a122f049a6e4791982a202390cb924f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a122f049a6e4791982a202390cb924f" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 39075 } }
I20260621 07:51:02.664129 19455 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a122f049a6e4791982a202390cb924f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:02.664178 19455 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a122f049a6e4791982a202390cb924f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a122f049a6e4791982a202390cb924f, State: Initialized, Role: FOLLOWER
I20260621 07:51:02.664641 19455 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a122f049a6e4791982a202390cb924f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a122f049a6e4791982a202390cb924f" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 39075 } }
I20260621 07:51:02.664793 19455 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a122f049a6e4791982a202390cb924f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:51:02.664855 19455 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a122f049a6e4791982a202390cb924f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:51:02.664947 19455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a122f049a6e4791982a202390cb924f [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:02.665917 19455 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a122f049a6e4791982a202390cb924f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a122f049a6e4791982a202390cb924f" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 39075 } }
I20260621 07:51:02.666226 19455 leader_election.cc:302] T 00000000000000000000000000000000 P 5a122f049a6e4791982a202390cb924f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a122f049a6e4791982a202390cb924f; no voters:
I20260621 07:51:02.666493 19455 leader_election.cc:288] T 00000000000000000000000000000000 P 5a122f049a6e4791982a202390cb924f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:51:02.666546 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:02.659628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44426 (local address 127.0.192.254:39075)
0621 07:51:02.660221 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:02.660236 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:02.660266 (+ 30us) server_negotiation.cc:407] Connection header received
0621 07:51:02.660867 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:02.660882 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:02.661150 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:02.661475 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:02.661851 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:02.662659 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:02.663455 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:02.663846 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:02.663992 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:02.664740 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:02.664747 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:02.664762 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:51:02.665204 (+ 442us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:02.665264 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:51:02.665285 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:02.665384 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:02.665498 (+ 114us) server_negotiation.cc:299] Negotiation successful
0621 07:51:02.665745 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":119,"threads_started":1}
I20260621 07:51:02.666585 19460 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a122f049a6e4791982a202390cb924f [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:02.666806 19460 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a122f049a6e4791982a202390cb924f [term 1 LEADER]: Becoming Leader. State: Replica: 5a122f049a6e4791982a202390cb924f, State: Running, Role: LEADER
I20260621 07:51:02.667071 19460 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a122f049a6e4791982a202390cb924f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a122f049a6e4791982a202390cb924f" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 39075 } }
I20260621 07:51:02.667635 19455 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a122f049a6e4791982a202390cb924f [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:51:02.668174 19462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a122f049a6e4791982a202390cb924f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a122f049a6e4791982a202390cb924f. Latest consensus state: current_term: 1 leader_uuid: "5a122f049a6e4791982a202390cb924f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a122f049a6e4791982a202390cb924f" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 39075 } } }
I20260621 07:51:02.668296 19462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a122f049a6e4791982a202390cb924f [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:02.668156 19461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a122f049a6e4791982a202390cb924f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a122f049a6e4791982a202390cb924f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a122f049a6e4791982a202390cb924f" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 39075 } } }
I20260621 07:51:02.668519 19461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a122f049a6e4791982a202390cb924f [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:02.668792 19467 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:51:02.671339 19467 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:51:02.675770 19467 catalog_manager.cc:1347] Generated new cluster ID: 04735543aa044a4e8592fe9ea947926c
I20260621 07:51:02.675832 19467 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:51:02.685585 19467 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:51:02.686241 19467 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:51:02.694895 19467 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a122f049a6e4791982a202390cb924f: Generated new TSK 0
I20260621 07:51:02.695444 19467 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:51:02.703513 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.193:0
--local_ip_for_outbound_sockets=127.0.192.193
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:39075
--builtin_ntp_servers=127.0.192.212:36153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 07:51:02.798853 19479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:02.799070 19479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:02.799125 19479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:02.802412 19479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:02.802474 19479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:02.802556 19479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.193
I20260621 07:51:02.806442 19479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:36153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:39075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.193
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:02.807302 19479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:02.808554 19479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 07:51:02.816797 19479 server_base.cc:1034] running on GCE node
W20260621 07:51:02.816756 19485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:02.816758 19487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:02.816738 19484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:02.817258 19479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:02.817822 19479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:02.818970 19479 hybrid_clock.cc:648] HybridClock initialized: now 1782028262818960 us; error 24 us; skew 500 ppm
I20260621 07:51:02.820433 19479 webserver.cc:460] Webserver started at http://127.0.192.193:44779/ using document root <none> and password file <none>
I20260621 07:51:02.820933 19479 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:02.820986 19479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:02.821183 19479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:02.822738 19479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-0/data/instance:
uuid: "28b6f14843ef490a82d4fc2ad768cbb7"
format_stamp: "Formatted at 2026-06-21 07:51:02 on dist-test-slave-lzx3"
server_key: "5d1f0e93b31c21408e0ff23deed43d39"
server_key_iv: "8c2b381042059db954aea991b06816ec"
server_key_version: "encryptionkey@0"
I20260621 07:51:02.823189 19479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-0/wal/instance:
uuid: "28b6f14843ef490a82d4fc2ad768cbb7"
format_stamp: "Formatted at 2026-06-21 07:51:02 on dist-test-slave-lzx3"
server_key: "5d1f0e93b31c21408e0ff23deed43d39"
server_key_iv: "8c2b381042059db954aea991b06816ec"
server_key_version: "encryptionkey@0"
I20260621 07:51:02.826079 19479 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 07:51:02.828106 19493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:02.829195 19479 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:51:02.829298 19479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-0/wal
uuid: "28b6f14843ef490a82d4fc2ad768cbb7"
format_stamp: "Formatted at 2026-06-21 07:51:02 on dist-test-slave-lzx3"
server_key: "5d1f0e93b31c21408e0ff23deed43d39"
server_key_iv: "8c2b381042059db954aea991b06816ec"
server_key_version: "encryptionkey@0"
I20260621 07:51:02.829384 19479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:02.843755 19479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:02.844259 19479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:02.844414 19479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:02.844937 19479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:51:02.845670 19479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:51:02.845709 19479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:02.845764 19479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:51:02.845793 19479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:02.853368 19479 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.193:34977
I20260621 07:51:02.853406 19606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.193:34977 every 8 connection(s)
I20260621 07:51:02.854080 19479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
I20260621 07:51:02.858636 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 19479
I20260621 07:51:02.858736 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-0/wal/instance
I20260621 07:51:02.859019 771 external_mini_cluster.cc:1428] Setting key 773524b999360b6aa425d817c4fe1713
I20260621 07:51:02.860853 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.194:0
--local_ip_for_outbound_sockets=127.0.192.194
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:39075
--builtin_ntp_servers=127.0.192.212:36153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 07:51:02.863036 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:02.855597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:52819 (local address 127.0.192.254:39075)
0621 07:51:02.855751 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:02.855755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:02.856599 (+ 844us) server_negotiation.cc:407] Connection header received
0621 07:51:02.857907 (+ 1308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:02.857910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:02.857969 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:02.858078 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:02.859018 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:02.859726 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:02.861134 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:02.861338 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:02.861416 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:02.862548 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:02.862553 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:02.862558 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:51:02.862572 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:02.862595 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:51:02.862604 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:02.862676 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:02.862795 (+ 119us) server_negotiation.cc:299] Negotiation successful
0621 07:51:02.862849 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 07:51:02.863623 19609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:02.855913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:39075 (local address 127.0.192.193:52819)
0621 07:51:02.856460 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:02.856492 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:02.857694 (+ 1202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:02.858238 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:02.858244 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:02.858347 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:02.858880 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:02.858895 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:02.859834 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:02.859836 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:02.860998 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:02.861011 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:02.861474 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:02.861480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:02.861684 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:02.862313 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:51:02.862314 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:02.862347 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:51:02.862399 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:02.862680 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:02.862685 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:02.862688 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 07:51:02.862764 (+ 76us) client_negotiation.cc:240] Negotiation successful
0621 07:51:02.862940 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":94,"threads_started":1}
I20260621 07:51:02.864738 19607 heartbeater.cc:347] Connected to a master server at 127.0.192.254:39075
I20260621 07:51:02.864984 19607 heartbeater.cc:464] Registering TS with master...
I20260621 07:51:02.865520 19607 heartbeater.cc:511] Master 127.0.192.254:39075 requested a full tablet report, sending...
I20260621 07:51:02.866916 19419 ts_manager.cc:194] Registered new tserver with Master: 28b6f14843ef490a82d4fc2ad768cbb7 (127.0.192.193:34977)
I20260621 07:51:02.868052 19419 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.192.193:52819
W20260621 07:51:02.960418 19610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:02.960691 19610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:02.960753 19610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:02.964329 19610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:02.964371 19610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:02.964417 19610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.194
I20260621 07:51:02.969045 19610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:36153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:39075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.194
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:02.969933 19610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:02.971145 19610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:02.980044 19616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:02.980051 19615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:02.980194 19610 server_base.cc:1034] running on GCE node
W20260621 07:51:02.980062 19618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:02.980772 19610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:02.981382 19610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:02.982558 19610 hybrid_clock.cc:648] HybridClock initialized: now 1782028262982546 us; error 25 us; skew 500 ppm
I20260621 07:51:02.984192 19610 webserver.cc:460] Webserver started at http://127.0.192.194:35075/ using document root <none> and password file <none>
I20260621 07:51:02.984766 19610 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:02.984853 19610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:02.985085 19610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:02.986913 19610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-1/data/instance:
uuid: "be7b2dd7cbf04e229ed8a557c7c0d1bf"
format_stamp: "Formatted at 2026-06-21 07:51:02 on dist-test-slave-lzx3"
server_key: "3fec9369f18d0cf271db408d5de850aa"
server_key_iv: "3b035b9af6406f9772420c975aca2eae"
server_key_version: "encryptionkey@0"
I20260621 07:51:02.987450 19610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-1/wal/instance:
uuid: "be7b2dd7cbf04e229ed8a557c7c0d1bf"
format_stamp: "Formatted at 2026-06-21 07:51:02 on dist-test-slave-lzx3"
server_key: "3fec9369f18d0cf271db408d5de850aa"
server_key_iv: "3b035b9af6406f9772420c975aca2eae"
server_key_version: "encryptionkey@0"
I20260621 07:51:02.990725 19610 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 07:51:02.992846 19624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:02.993919 19610 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260621 07:51:02.994042 19610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-1/wal
uuid: "be7b2dd7cbf04e229ed8a557c7c0d1bf"
format_stamp: "Formatted at 2026-06-21 07:51:02 on dist-test-slave-lzx3"
server_key: "3fec9369f18d0cf271db408d5de850aa"
server_key_iv: "3b035b9af6406f9772420c975aca2eae"
server_key_version: "encryptionkey@0"
I20260621 07:51:02.994141 19610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:03.008057 19610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:03.008661 19610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:03.008821 19610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:03.009333 19610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:51:03.010154 19610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:51:03.010221 19610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:03.010285 19610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:51:03.010331 19610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:03.018260 19610 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.194:45617
I20260621 07:51:03.018342 19737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.194:45617 every 8 connection(s)
I20260621 07:51:03.019054 19610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
I20260621 07:51:03.026708 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 19610
I20260621 07:51:03.026898 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-1/wal/instance
I20260621 07:51:03.027145 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:03.020669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:60659 (local address 127.0.192.254:39075)
0621 07:51:03.020789 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:03.020792 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:03.021752 (+ 960us) server_negotiation.cc:407] Connection header received
0621 07:51:03.022957 (+ 1205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:03.022960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:03.023006 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:03.023077 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:03.023818 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:03.024318 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:03.025388 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:03.025591 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:03.025674 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:03.026601 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:03.026606 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:03.026609 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:51:03.026626 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:03.026655 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:51:03.026669 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:03.026769 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:03.026945 (+ 176us) server_negotiation.cc:299] Negotiation successful
0621 07:51:03.027007 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260621 07:51:03.027233 771 external_mini_cluster.cc:1428] Setting key 15c6b943dba726d85bf16aa777c27a80
I20260621 07:51:03.027951 19740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:03.021040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:39075 (local address 127.0.192.194:60659)
0621 07:51:03.021604 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:03.021639 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:03.022769 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:03.023218 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:03.023224 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:03.023297 (+ 73us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:03.023698 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:03.023708 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:03.024415 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:03.024417 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:03.025287 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:03.025296 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:03.025712 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:03.025715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:03.025843 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:03.026324 (+ 481us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:51:03.026325 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:03.026358 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:51:03.026410 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:03.026764 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:03.026770 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:03.026773 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 07:51:03.026935 (+ 162us) client_negotiation.cc:240] Negotiation successful
0621 07:51:03.027170 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":103,"threads_started":1}
I20260621 07:51:03.028961 19738 heartbeater.cc:347] Connected to a master server at 127.0.192.254:39075
I20260621 07:51:03.029192 19738 heartbeater.cc:464] Registering TS with master...
I20260621 07:51:03.029399 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.195:0
--local_ip_for_outbound_sockets=127.0.192.195
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:39075
--builtin_ntp_servers=127.0.192.212:36153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 07:51:03.029759 19738 heartbeater.cc:511] Master 127.0.192.254:39075 requested a full tablet report, sending...
I20260621 07:51:03.030746 19418 ts_manager.cc:194] Registered new tserver with Master: be7b2dd7cbf04e229ed8a557c7c0d1bf (127.0.192.194:45617)
I20260621 07:51:03.031270 19418 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.192.194:60659
W20260621 07:51:03.130097 19741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:03.130331 19741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:03.130398 19741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:03.134194 19741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:03.134287 19741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:03.134394 19741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.195
I20260621 07:51:03.138970 19741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:36153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:39075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.195
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:03.139894 19741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:03.141184 19741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:03.149401 19746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:03.149444 19747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:03.149461 19741 server_base.cc:1034] running on GCE node
W20260621 07:51:03.149401 19749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:03.149902 19741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:03.150394 19741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:03.151535 19741 hybrid_clock.cc:648] HybridClock initialized: now 1782028263151525 us; error 26 us; skew 500 ppm
I20260621 07:51:03.152987 19741 webserver.cc:460] Webserver started at http://127.0.192.195:43523/ using document root <none> and password file <none>
I20260621 07:51:03.153399 19741 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:03.153445 19741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:03.153573 19741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:03.155226 19741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-2/data/instance:
uuid: "6df958c54f9a46068c6e75db8ff8aaf0"
format_stamp: "Formatted at 2026-06-21 07:51:03 on dist-test-slave-lzx3"
server_key: "4e3a112888be3621f45e20bbf7ef61c9"
server_key_iv: "cb98e759333269638d3fd2a2808de31d"
server_key_version: "encryptionkey@0"
I20260621 07:51:03.155649 19741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-2/wal/instance:
uuid: "6df958c54f9a46068c6e75db8ff8aaf0"
format_stamp: "Formatted at 2026-06-21 07:51:03 on dist-test-slave-lzx3"
server_key: "4e3a112888be3621f45e20bbf7ef61c9"
server_key_iv: "cb98e759333269638d3fd2a2808de31d"
server_key_version: "encryptionkey@0"
I20260621 07:51:03.158715 19741 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 07:51:03.160547 19755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:03.161545 19741 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:51:03.161746 19741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-2/wal
uuid: "6df958c54f9a46068c6e75db8ff8aaf0"
format_stamp: "Formatted at 2026-06-21 07:51:03 on dist-test-slave-lzx3"
server_key: "4e3a112888be3621f45e20bbf7ef61c9"
server_key_iv: "cb98e759333269638d3fd2a2808de31d"
server_key_version: "encryptionkey@0"
I20260621 07:51:03.161835 19741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:03.175539 19741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:03.176023 19741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:03.176169 19741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:03.176725 19741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:51:03.177524 19741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:51:03.177567 19741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:03.177603 19741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:51:03.177618 19741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:03.185623 19741 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.195:39685
I20260621 07:51:03.185649 19868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.195:39685 every 8 connection(s)
I20260621 07:51:03.186298 19741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
I20260621 07:51:03.193635 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:03.187844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:37343 (local address 127.0.192.254:39075)
0621 07:51:03.187993 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:03.187997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:03.188948 (+ 951us) server_negotiation.cc:407] Connection header received
0621 07:51:03.190089 (+ 1141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:03.190091 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:03.190131 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:03.190196 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:03.190898 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:03.191320 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:03.192215 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:03.192392 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:03.192455 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:03.193303 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:03.193306 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:03.193307 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:51:03.193317 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:03.193334 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:51:03.193341 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:03.193391 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:03.193503 (+ 112us) server_negotiation.cc:299] Negotiation successful
0621 07:51:03.193542 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 07:51:03.194275 19871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:03.188147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:39075 (local address 127.0.192.195:37343)
0621 07:51:03.188814 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:03.188846 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:03.189917 (+ 1071us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:03.190332 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:03.190338 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:03.190406 (+ 68us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:03.190796 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:03.190806 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:03.191412 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:03.191414 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:03.192125 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:03.192133 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:03.192497 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:03.192499 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:03.192667 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:03.193101 (+ 434us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:51:03.193102 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:03.193133 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:51:03.193182 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:03.193404 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:03.193410 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:03.193413 (+ 3us) client_negotiation.cc:769] Sending connection context
0621 07:51:03.193488 (+ 75us) client_negotiation.cc:240] Negotiation successful
0621 07:51:03.193650 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":117,"threads_started":1}
I20260621 07:51:03.195215 19869 heartbeater.cc:347] Connected to a master server at 127.0.192.254:39075
I20260621 07:51:03.195353 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 19741
I20260621 07:51:03.195415 19869 heartbeater.cc:464] Registering TS with master...
I20260621 07:51:03.195468 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-2/wal/instance
I20260621 07:51:03.195703 771 external_mini_cluster.cc:1428] Setting key 64103b02a2941c0bde740a91ddc54be3
I20260621 07:51:03.195886 19869 heartbeater.cc:511] Master 127.0.192.254:39075 requested a full tablet report, sending...
I20260621 07:51:03.196789 19418 ts_manager.cc:194] Registered new tserver with Master: 6df958c54f9a46068c6e75db8ff8aaf0 (127.0.192.195:39685)
I20260621 07:51:03.197341 19418 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.192.195:37343
I20260621 07:51:03.197736 771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:51:03.202394 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:03.199511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44442 (local address 127.0.192.254:39075)
0621 07:51:03.199613 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:03.199615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:03.199787 (+ 172us) server_negotiation.cc:407] Connection header received
0621 07:51:03.199877 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:03.199878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:03.199908 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:03.199964 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:03.200168 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:03.200655 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:03.201375 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:03.201537 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:03.201597 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:03.202131 (+ 534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:03.202134 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:03.202135 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:51:03.202144 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:03.202157 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:51:03.202163 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:03.202245 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:03.202293 (+ 48us) server_negotiation.cc:299] Negotiation successful
0621 07:51:03.202316 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260621 07:51:03.204701 19418 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44442:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:51:03.207252 19418 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:51:03.219901 19883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:03.214780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47010 (local address 127.0.192.195:39685)
0621 07:51:03.215265 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:03.215279 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:03.215300 (+ 21us) server_negotiation.cc:407] Connection header received
0621 07:51:03.215351 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:03.215354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:03.215526 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:03.215659 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:51:03.216046 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:03.217091 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:03.218814 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:03.219280 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:03.219395 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:03.219552 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:03.219687 (+ 135us) server_negotiation.cc:299] Negotiation successful
0621 07:51:03.219771 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":111,"threads_started":1}
I20260621 07:51:03.219897 19881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:03.214366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.195:39685 (local address 127.0.0.1:47010)
0621 07:51:03.214803 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:03.214817 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:03.214908 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:03.215786 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:03.215791 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:03.215811 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:51:03.215932 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:03.215939 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:03.217254 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:03.217257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:03.218700 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:03.218709 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:03.219395 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:03.219398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:03.219502 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:03.219535 (+ 33us) client_negotiation.cc:769] Sending connection context
0621 07:51:03.219661 (+ 126us) client_negotiation.cc:240] Negotiation successful
0621 07:51:03.219763 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":77,"thread_start_us":85,"threads_started":1}
I20260621 07:51:03.220276 19880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:03.214219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:34977 (local address 127.0.0.1:52240)
0621 07:51:03.214521 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:03.214564 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:03.214678 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:03.215918 (+ 1240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:03.215921 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:03.215942 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:51:03.216025 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:03.216030 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:03.217319 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:03.217322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:03.218758 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:03.218765 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:03.219716 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:03.219719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:03.219897 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:03.219954 (+ 57us) client_negotiation.cc:769] Sending connection context
0621 07:51:03.220093 (+ 139us) client_negotiation.cc:240] Negotiation successful
0621 07:51:03.220149 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":260,"thread_start_us":69,"threads_started":1}
I20260621 07:51:03.220498 19884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:03.215218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:45617 (local address 127.0.0.1:51646)
0621 07:51:03.215472 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:03.215483 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:03.215556 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:03.216318 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:03.216320 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:03.216336 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:51:03.216405 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:03.216410 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:03.217376 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:03.217379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:03.218755 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:03.218765 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:03.219716 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:03.219719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:03.219898 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:03.220203 (+ 305us) client_negotiation.cc:769] Sending connection context
0621 07:51:03.220252 (+ 49us) client_negotiation.cc:240] Negotiation successful
0621 07:51:03.220311 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":362,"thread_start_us":78,"threads_started":1}
I20260621 07:51:03.220903 19882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:03.214340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52240 (local address 127.0.192.193:34977)
0621 07:51:03.215138 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:03.215148 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:03.215171 (+ 23us) server_negotiation.cc:407] Connection header received
0621 07:51:03.215242 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:03.215248 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:03.215786 (+ 538us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:03.215927 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:51:03.216129 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:03.217142 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:03.218885 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:03.219585 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:03.220426 (+ 841us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:03.220617 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:03.220664 (+ 47us) server_negotiation.cc:299] Negotiation successful
0621 07:51:03.220743 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":129,"threads_started":1}
I20260621 07:51:03.221139 19885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:03.215356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51646 (local address 127.0.192.194:45617)
0621 07:51:03.215918 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:03.215929 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:03.215949 (+ 20us) server_negotiation.cc:407] Connection header received
0621 07:51:03.216017 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:03.216022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:03.216190 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:03.216328 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:51:03.216525 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:03.217277 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:03.218885 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:03.219585 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:03.220705 (+ 1120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:03.220864 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:03.220910 (+ 46us) server_negotiation.cc:299] Negotiation successful
0621 07:51:03.221000 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":111,"threads_started":1}
I20260621 07:51:03.222759 19803 tablet_service.cc:1467] Processing CreateTablet for tablet 1e7aa8a673ec41e4bb3cd50e6b2be354 (DEFAULT_TABLE table=test-table [id=90bed5547acf4d5f9373dd04c76bc0d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:51:03.223148 19672 tablet_service.cc:1467] Processing CreateTablet for tablet 1e7aa8a673ec41e4bb3cd50e6b2be354 (DEFAULT_TABLE table=test-table [id=90bed5547acf4d5f9373dd04c76bc0d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:51:03.223307 19541 tablet_service.cc:1467] Processing CreateTablet for tablet 1e7aa8a673ec41e4bb3cd50e6b2be354 (DEFAULT_TABLE table=test-table [id=90bed5547acf4d5f9373dd04c76bc0d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:51:03.223749 19803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e7aa8a673ec41e4bb3cd50e6b2be354. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:03.223960 19672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e7aa8a673ec41e4bb3cd50e6b2be354. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:03.224134 19541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e7aa8a673ec41e4bb3cd50e6b2be354. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:03.229446 19886 tablet_bootstrap.cc:492] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P be7b2dd7cbf04e229ed8a557c7c0d1bf: Bootstrap starting.
I20260621 07:51:03.231812 19886 tablet_bootstrap.cc:654] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P be7b2dd7cbf04e229ed8a557c7c0d1bf: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:03.232501 19886 log.cc:824] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P be7b2dd7cbf04e229ed8a557c7c0d1bf: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:03.233515 19888 tablet_bootstrap.cc:492] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P 28b6f14843ef490a82d4fc2ad768cbb7: Bootstrap starting.
I20260621 07:51:03.234999 19886 tablet_bootstrap.cc:492] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P be7b2dd7cbf04e229ed8a557c7c0d1bf: No bootstrap required, opened a new log
I20260621 07:51:03.235188 19886 ts_tablet_manager.cc:1397] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P be7b2dd7cbf04e229ed8a557c7c0d1bf: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260621 07:51:03.235661 19888 tablet_bootstrap.cc:654] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P 28b6f14843ef490a82d4fc2ad768cbb7: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:03.236452 19888 log.cc:824] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P 28b6f14843ef490a82d4fc2ad768cbb7: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:03.233145 19887 tablet_bootstrap.cc:492] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P 6df958c54f9a46068c6e75db8ff8aaf0: Bootstrap starting.
I20260621 07:51:03.238875 19887 tablet_bootstrap.cc:654] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P 6df958c54f9a46068c6e75db8ff8aaf0: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:03.239094 19888 tablet_bootstrap.cc:492] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P 28b6f14843ef490a82d4fc2ad768cbb7: No bootstrap required, opened a new log
I20260621 07:51:03.239320 19888 ts_tablet_manager.cc:1397] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P 28b6f14843ef490a82d4fc2ad768cbb7: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260621 07:51:03.239202 19886 raft_consensus.cc:348] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P be7b2dd7cbf04e229ed8a557c7c0d1bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b6f14843ef490a82d4fc2ad768cbb7" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 34977 } } peers { permanent_uuid: "6df958c54f9a46068c6e75db8ff8aaf0" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 39685 } } peers { permanent_uuid: "be7b2dd7cbf04e229ed8a557c7c0d1bf" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 45617 } }
I20260621 07:51:03.239447 19886 raft_consensus.cc:374] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P be7b2dd7cbf04e229ed8a557c7c0d1bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:03.239501 19886 raft_consensus.cc:729] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P be7b2dd7cbf04e229ed8a557c7c0d1bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be7b2dd7cbf04e229ed8a557c7c0d1bf, State: Initialized, Role: FOLLOWER
I20260621 07:51:03.239660 19887 log.cc:824] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P 6df958c54f9a46068c6e75db8ff8aaf0: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:03.239984 19886 consensus_queue.cc:260] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P be7b2dd7cbf04e229ed8a557c7c0d1bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b6f14843ef490a82d4fc2ad768cbb7" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 34977 } } peers { permanent_uuid: "6df958c54f9a46068c6e75db8ff8aaf0" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 39685 } } peers { permanent_uuid: "be7b2dd7cbf04e229ed8a557c7c0d1bf" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 45617 } }
I20260621 07:51:03.241016 19886 ts_tablet_manager.cc:1428] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P be7b2dd7cbf04e229ed8a557c7c0d1bf: Time spent starting tablet: real 0.006s user 0.005s sys 0.003s
I20260621 07:51:03.241546 19887 tablet_bootstrap.cc:492] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P 6df958c54f9a46068c6e75db8ff8aaf0: No bootstrap required, opened a new log
I20260621 07:51:03.241675 19887 ts_tablet_manager.cc:1397] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P 6df958c54f9a46068c6e75db8ff8aaf0: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260621 07:51:03.242630 19888 raft_consensus.cc:348] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P 28b6f14843ef490a82d4fc2ad768cbb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b6f14843ef490a82d4fc2ad768cbb7" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 34977 } } peers { permanent_uuid: "6df958c54f9a46068c6e75db8ff8aaf0" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 39685 } } peers { permanent_uuid: "be7b2dd7cbf04e229ed8a557c7c0d1bf" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 45617 } }
I20260621 07:51:03.242805 19888 raft_consensus.cc:374] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P 28b6f14843ef490a82d4fc2ad768cbb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:03.242836 19888 raft_consensus.cc:729] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P 28b6f14843ef490a82d4fc2ad768cbb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b6f14843ef490a82d4fc2ad768cbb7, State: Initialized, Role: FOLLOWER
I20260621 07:51:03.243189 19888 consensus_queue.cc:260] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P 28b6f14843ef490a82d4fc2ad768cbb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b6f14843ef490a82d4fc2ad768cbb7" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 34977 } } peers { permanent_uuid: "6df958c54f9a46068c6e75db8ff8aaf0" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 39685 } } peers { permanent_uuid: "be7b2dd7cbf04e229ed8a557c7c0d1bf" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 45617 } }
I20260621 07:51:03.243853 19607 heartbeater.cc:503] Master 127.0.192.254:39075 was elected leader, sending a full tablet report...
I20260621 07:51:03.244035 19888 ts_tablet_manager.cc:1428] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P 28b6f14843ef490a82d4fc2ad768cbb7: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260621 07:51:03.244728 19738 heartbeater.cc:503] Master 127.0.192.254:39075 was elected leader, sending a full tablet report...
I20260621 07:51:03.244880 19887 raft_consensus.cc:348] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P 6df958c54f9a46068c6e75db8ff8aaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b6f14843ef490a82d4fc2ad768cbb7" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 34977 } } peers { permanent_uuid: "6df958c54f9a46068c6e75db8ff8aaf0" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 39685 } } peers { permanent_uuid: "be7b2dd7cbf04e229ed8a557c7c0d1bf" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 45617 } }
I20260621 07:51:03.245126 19887 raft_consensus.cc:374] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P 6df958c54f9a46068c6e75db8ff8aaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:03.245177 19887 raft_consensus.cc:729] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P 6df958c54f9a46068c6e75db8ff8aaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6df958c54f9a46068c6e75db8ff8aaf0, State: Initialized, Role: FOLLOWER
I20260621 07:51:03.245505 19887 consensus_queue.cc:260] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P 6df958c54f9a46068c6e75db8ff8aaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b6f14843ef490a82d4fc2ad768cbb7" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 34977 } } peers { permanent_uuid: "6df958c54f9a46068c6e75db8ff8aaf0" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 39685 } } peers { permanent_uuid: "be7b2dd7cbf04e229ed8a557c7c0d1bf" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 45617 } }
I20260621 07:51:03.246141 19869 heartbeater.cc:503] Master 127.0.192.254:39075 was elected leader, sending a full tablet report...
I20260621 07:51:03.246367 19887 ts_tablet_manager.cc:1428] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P 6df958c54f9a46068c6e75db8ff8aaf0: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260621 07:51:03.247852 19892 raft_consensus.cc:482] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P be7b2dd7cbf04e229ed8a557c7c0d1bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:51:03.247988 19892 raft_consensus.cc:504] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P be7b2dd7cbf04e229ed8a557c7c0d1bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b6f14843ef490a82d4fc2ad768cbb7" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 34977 } } peers { permanent_uuid: "6df958c54f9a46068c6e75db8ff8aaf0" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 39685 } } peers { permanent_uuid: "be7b2dd7cbf04e229ed8a557c7c0d1bf" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 45617 } }
I20260621 07:51:03.249130 19892 leader_election.cc:288] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P be7b2dd7cbf04e229ed8a557c7c0d1bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28b6f14843ef490a82d4fc2ad768cbb7 (127.0.192.193:34977), 6df958c54f9a46068c6e75db8ff8aaf0 (127.0.192.195:39685)
I20260621 07:51:03.252945 19883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:03.249740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:48871 (local address 127.0.192.195:39685)
0621 07:51:03.249859 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:03.249865 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:03.249879 (+ 14us) server_negotiation.cc:407] Connection header received
0621 07:51:03.249918 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:03.249921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:03.249958 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:03.250052 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:51:03.250337 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:03.250819 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:03.251863 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:03.252248 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:03.252371 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:03.252421 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:03.252754 (+ 333us) server_negotiation.cc:299] Negotiation successful
0621 07:51:03.252826 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260621 07:51:03.252951 19740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:03.249344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:34977 (local address 127.0.192.194:49391)
0621 07:51:03.249495 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:03.249509 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:03.249586 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:03.249904 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:03.249907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:03.249927 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:51:03.250066 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:03.250075 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:03.250833 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:03.250838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:03.251739 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:03.251749 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:03.252367 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:03.252371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:03.252533 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:03.252671 (+ 138us) client_negotiation.cc:769] Sending connection context
0621 07:51:03.252765 (+ 94us) client_negotiation.cc:240] Negotiation successful
0621 07:51:03.252831 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":102}
I20260621 07:51:03.252954 19895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:03.249388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.195:39685 (local address 127.0.192.194:48871)
0621 07:51:03.249693 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:03.249706 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:03.249774 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:03.250067 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:03.250069 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:03.250082 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:51:03.250222 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:03.250229 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:03.250930 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:03.250933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:03.251741 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:03.251749 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:03.252373 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:03.252376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:03.252516 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:03.252657 (+ 141us) client_negotiation.cc:769] Sending connection context
0621 07:51:03.252716 (+ 59us) client_negotiation.cc:240] Negotiation successful
0621 07:51:03.252800 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":161,"thread_start_us":115,"threads_started":1}
I20260621 07:51:03.252959 19882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:03.249383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:49391 (local address 127.0.192.193:34977)
0621 07:51:03.249491 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:03.249495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:03.249708 (+ 213us) server_negotiation.cc:407] Connection header received
0621 07:51:03.249755 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:03.249758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:03.249799 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:03.249866 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:51:03.250205 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:03.250727 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:03.251891 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:03.252253 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:03.252372 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:03.252418 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:03.252800 (+ 382us) server_negotiation.cc:299] Negotiation successful
0621 07:51:03.252868 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260621 07:51:03.253634 19823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e7aa8a673ec41e4bb3cd50e6b2be354" candidate_uuid: "be7b2dd7cbf04e229ed8a557c7c0d1bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6df958c54f9a46068c6e75db8ff8aaf0" is_pre_election: true
I20260621 07:51:03.253715 19561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e7aa8a673ec41e4bb3cd50e6b2be354" candidate_uuid: "be7b2dd7cbf04e229ed8a557c7c0d1bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28b6f14843ef490a82d4fc2ad768cbb7" is_pre_election: true
I20260621 07:51:03.253877 19823 raft_consensus.cc:2454] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P 6df958c54f9a46068c6e75db8ff8aaf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be7b2dd7cbf04e229ed8a557c7c0d1bf in term 0.
I20260621 07:51:03.253897 19561 raft_consensus.cc:2454] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P 28b6f14843ef490a82d4fc2ad768cbb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be7b2dd7cbf04e229ed8a557c7c0d1bf in term 0.
I20260621 07:51:03.254287 19625 leader_election.cc:302] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P be7b2dd7cbf04e229ed8a557c7c0d1bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28b6f14843ef490a82d4fc2ad768cbb7, be7b2dd7cbf04e229ed8a557c7c0d1bf; no voters:
I20260621 07:51:03.254478 19892 raft_consensus.cc:2781] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P be7b2dd7cbf04e229ed8a557c7c0d1bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:51:03.254549 19892 raft_consensus.cc:482] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P be7b2dd7cbf04e229ed8a557c7c0d1bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:51:03.254580 19892 raft_consensus.cc:3037] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P be7b2dd7cbf04e229ed8a557c7c0d1bf [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:03.255645 19892 raft_consensus.cc:504] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P be7b2dd7cbf04e229ed8a557c7c0d1bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b6f14843ef490a82d4fc2ad768cbb7" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 34977 } } peers { permanent_uuid: "6df958c54f9a46068c6e75db8ff8aaf0" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 39685 } } peers { permanent_uuid: "be7b2dd7cbf04e229ed8a557c7c0d1bf" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 45617 } }
I20260621 07:51:03.255954 19892 leader_election.cc:288] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P be7b2dd7cbf04e229ed8a557c7c0d1bf [CANDIDATE]: Term 1 election: Requested vote from peers 28b6f14843ef490a82d4fc2ad768cbb7 (127.0.192.193:34977), 6df958c54f9a46068c6e75db8ff8aaf0 (127.0.192.195:39685)
I20260621 07:51:03.256273 19823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e7aa8a673ec41e4bb3cd50e6b2be354" candidate_uuid: "be7b2dd7cbf04e229ed8a557c7c0d1bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6df958c54f9a46068c6e75db8ff8aaf0"
I20260621 07:51:03.256292 19561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e7aa8a673ec41e4bb3cd50e6b2be354" candidate_uuid: "be7b2dd7cbf04e229ed8a557c7c0d1bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28b6f14843ef490a82d4fc2ad768cbb7"
I20260621 07:51:03.256399 19561 raft_consensus.cc:3037] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P 28b6f14843ef490a82d4fc2ad768cbb7 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:03.256390 19823 raft_consensus.cc:3037] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P 6df958c54f9a46068c6e75db8ff8aaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:03.257545 19823 raft_consensus.cc:2454] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P 6df958c54f9a46068c6e75db8ff8aaf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be7b2dd7cbf04e229ed8a557c7c0d1bf in term 1.
I20260621 07:51:03.257545 19561 raft_consensus.cc:2454] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P 28b6f14843ef490a82d4fc2ad768cbb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be7b2dd7cbf04e229ed8a557c7c0d1bf in term 1.
I20260621 07:51:03.257815 19625 leader_election.cc:302] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P be7b2dd7cbf04e229ed8a557c7c0d1bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28b6f14843ef490a82d4fc2ad768cbb7, be7b2dd7cbf04e229ed8a557c7c0d1bf; no voters:
I20260621 07:51:03.258062 19892 raft_consensus.cc:2781] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P be7b2dd7cbf04e229ed8a557c7c0d1bf [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:03.258222 19892 raft_consensus.cc:686] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P be7b2dd7cbf04e229ed8a557c7c0d1bf [term 1 LEADER]: Becoming Leader. State: Replica: be7b2dd7cbf04e229ed8a557c7c0d1bf, State: Running, Role: LEADER
I20260621 07:51:03.258491 19892 consensus_queue.cc:237] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P be7b2dd7cbf04e229ed8a557c7c0d1bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b6f14843ef490a82d4fc2ad768cbb7" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 34977 } } peers { permanent_uuid: "6df958c54f9a46068c6e75db8ff8aaf0" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 39685 } } peers { permanent_uuid: "be7b2dd7cbf04e229ed8a557c7c0d1bf" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 45617 } }
I20260621 07:51:03.261080 19419 catalog_manager.cc:5495] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P be7b2dd7cbf04e229ed8a557c7c0d1bf reported cstate change: term changed from 0 to 1, leader changed from <none> to be7b2dd7cbf04e229ed8a557c7c0d1bf (127.0.192.194). New cstate: current_term: 1 leader_uuid: "be7b2dd7cbf04e229ed8a557c7c0d1bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28b6f14843ef490a82d4fc2ad768cbb7" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 34977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6df958c54f9a46068c6e75db8ff8aaf0" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 39685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be7b2dd7cbf04e229ed8a557c7c0d1bf" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 45617 } health_report { overall_health: HEALTHY } } }
W20260621 07:51:03.269961 19739 tablet.cc:2369] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P be7b2dd7cbf04e229ed8a557c7c0d1bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:51:03.272760 19885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:03.269798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51654 (local address 127.0.192.194:45617)
0621 07:51:03.269894 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:03.269898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:03.269914 (+ 16us) server_negotiation.cc:407] Connection header received
0621 07:51:03.270031 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:03.270034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:03.270085 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:03.270190 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:51:03.270425 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:03.270891 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:03.271603 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:03.271784 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:03.271853 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:03.272065 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:51:03.272383 (+ 318us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:51:03.272497 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:03.272633 (+ 136us) server_negotiation.cc:299] Negotiation successful
0621 07:51:03.272664 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260621 07:51:03.277387 19823 raft_consensus.cc:1261] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P 6df958c54f9a46068c6e75db8ff8aaf0 [term 1 FOLLOWER]: Refusing update from remote peer be7b2dd7cbf04e229ed8a557c7c0d1bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:51:03.277405 19561 raft_consensus.cc:1261] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P 28b6f14843ef490a82d4fc2ad768cbb7 [term 1 FOLLOWER]: Refusing update from remote peer be7b2dd7cbf04e229ed8a557c7c0d1bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:51:03.278048 19896 consensus_queue.cc:1035] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P be7b2dd7cbf04e229ed8a557c7c0d1bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "28b6f14843ef490a82d4fc2ad768cbb7" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 34977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:51:03.278226 19892 consensus_queue.cc:1035] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P be7b2dd7cbf04e229ed8a557c7c0d1bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6df958c54f9a46068c6e75db8ff8aaf0" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 39685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:51:03.285688 19900 mvcc.cc:205] Tried to move back new op lower bound from 7299187766379077632 to 7299187766309613568. Current Snapshot: MvccSnapshot[applied={T|T < 7299187766379077632}]
I20260621 07:51:03.285808 19901 mvcc.cc:205] Tried to move back new op lower bound from 7299187766379077632 to 7299187766309613568. Current Snapshot: MvccSnapshot[applied={T|T < 7299187766379077632}]
I20260621 07:51:03.291389 19419 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:51:03.291531 19419 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:51:03.293694 19419 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a122f049a6e4791982a202390cb924f: Sending DeleteTablet for 3 replicas of tablet 1e7aa8a673ec41e4bb3cd50e6b2be354
I20260621 07:51:03.294307 19541 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e7aa8a673ec41e4bb3cd50e6b2be354 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:51:03 UTC) from {username='slave'} at 127.0.0.1:52240
I20260621 07:51:03.294485 19803 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e7aa8a673ec41e4bb3cd50e6b2be354 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:51:03 UTC) from {username='slave'} at 127.0.0.1:47010
I20260621 07:51:03.294872 19909 tablet_replica.cc:330] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P 28b6f14843ef490a82d4fc2ad768cbb7: stopping tablet replica
I20260621 07:51:03.294878 19672 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e7aa8a673ec41e4bb3cd50e6b2be354 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:51:03 UTC) from {username='slave'} at 127.0.0.1:51646
I20260621 07:51:03.295153 19909 raft_consensus.cc:2229] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P 28b6f14843ef490a82d4fc2ad768cbb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:51:03.295249 19911 tablet_replica.cc:330] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P be7b2dd7cbf04e229ed8a557c7c0d1bf: stopping tablet replica
I20260621 07:51:03.295482 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 19479
I20260621 07:51:03.295470 19911 raft_consensus.cc:2229] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P be7b2dd7cbf04e229ed8a557c7c0d1bf [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:51:03.296947 19910 tablet_replica.cc:330] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P 6df958c54f9a46068c6e75db8ff8aaf0: stopping tablet replica
I20260621 07:51:03.297230 19910 raft_consensus.cc:2229] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P 6df958c54f9a46068c6e75db8ff8aaf0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:51:03.297488 19910 raft_consensus.cc:2258] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P 6df958c54f9a46068c6e75db8ff8aaf0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:51:03.297699 19911 raft_consensus.cc:2258] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P be7b2dd7cbf04e229ed8a557c7c0d1bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:51:03.299043 19911 ts_tablet_manager.cc:1905] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P be7b2dd7cbf04e229ed8a557c7c0d1bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:51:03.301620 19910 ts_tablet_manager.cc:1905] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P 6df958c54f9a46068c6e75db8ff8aaf0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:51:03.302335 19911 ts_tablet_manager.cc:1918] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P be7b2dd7cbf04e229ed8a557c7c0d1bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:51:03.302428 19911 log.cc:1196] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P be7b2dd7cbf04e229ed8a557c7c0d1bf: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-1/wal/wals/1e7aa8a673ec41e4bb3cd50e6b2be354
I20260621 07:51:03.302714 19911 ts_tablet_manager.cc:1939] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P be7b2dd7cbf04e229ed8a557c7c0d1bf: Deleting consensus metadata
I20260621 07:51:03.303701 19406 catalog_manager.cc:4841] TS be7b2dd7cbf04e229ed8a557c7c0d1bf (127.0.192.194:45617): tablet 1e7aa8a673ec41e4bb3cd50e6b2be354 (table test-table [id=90bed5547acf4d5f9373dd04c76bc0d0]) successfully deleted
I20260621 07:51:03.304109 19910 ts_tablet_manager.cc:1918] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P 6df958c54f9a46068c6e75db8ff8aaf0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260621 07:51:03.304180 19403 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.192.193:34977 (error 108)
I20260621 07:51:03.304246 19910 log.cc:1196] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P 6df958c54f9a46068c6e75db8ff8aaf0: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782028028522580-771-0/minicluster-data/ts-2/wal/wals/1e7aa8a673ec41e4bb3cd50e6b2be354
I20260621 07:51:03.304502 19910 ts_tablet_manager.cc:1939] T 1e7aa8a673ec41e4bb3cd50e6b2be354 P 6df958c54f9a46068c6e75db8ff8aaf0: Deleting consensus metadata
I20260621 07:51:03.304930 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 19610
I20260621 07:51:03.305194 19884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:03.304867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:34977 (local address 127.0.0.1:52250)
0621 07:51:03.305033 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:03.305099 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.193:34977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260621 07:51:03.305598 19406 catalog_manager.cc:4841] TS 6df958c54f9a46068c6e75db8ff8aaf0 (127.0.192.195:39685): tablet 1e7aa8a673ec41e4bb3cd50e6b2be354 (table test-table [id=90bed5547acf4d5f9373dd04c76bc0d0]) successfully deleted
W20260621 07:51:03.306051 19403 catalog_manager.cc:4584] TS 28b6f14843ef490a82d4fc2ad768cbb7 (127.0.192.193:34977): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1e7aa8a673ec41e4bb3cd50e6b2be354: Network error: Client connection negotiation failed: client connection to 127.0.192.193:34977: connect: Connection refused (error 111)
I20260621 07:51:03.310338 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 19741
I20260621 07:51:03.315521 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 19388
2026-06-21T07:51:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (837 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:51:03 dist-test-slave-lzx3 krb5kdc[19915](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:51:03 dist-test-slave-lzx3 krb5kdc[19915](info): set up 2 sockets
Jun 21 07:51:03 dist-test-slave-lzx3 krb5kdc[19915](info): commencing operation
krb5kdc: starting...
W20260621 07:51:04.252104 771 mini_kdc.cc:121] Time spent starting KDC: real 0.917s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/krb5kdc/test-user.keytab.
Jun 21 07:51:04 dist-test-slave-lzx3 krb5kdc[19915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:51:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:51:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.192.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
I20260621 07:51:04.352541 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:37995
--webserver_interface=127.0.192.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:45179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:37995
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:04.447402 19928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:04.447630 19928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:04.447669 19928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:04.451050 19928 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:51:04.451093 19928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:04.451110 19928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:04.451124 19928 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:51:04.451180 19928 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:51:04.455447 19928 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:45179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:37995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:37995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:04.456508 19928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:04.457511 19928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 07:51:04.464699 19928 server_base.cc:1034] running on GCE node
W20260621 07:51:04.464587 19934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:04.464582 19936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:04.464649 19933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:04.465323 19928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:04.466190 19928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:04.467345 19928 hybrid_clock.cc:648] HybridClock initialized: now 1782028264467323 us; error 44 us; skew 500 ppm
Jun 21 07:51:04 dist-test-slave-lzx3 krb5kdc[19915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028264, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:04.469985 19928 init.cc:377] Logged in from keytab as kudu/127.0.192.254@KRBTEST.COM (short username kudu)
I20260621 07:51:04.470700 19928 webserver.cc:460] Webserver started at http://127.0.192.254:45065/ using document root <none> and password file <none>
I20260621 07:51:04.471117 19928 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:04.471166 19928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:04.471299 19928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:04.472985 19928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "3ad41e6ee44d4216b55a1c823ea86682"
format_stamp: "Formatted at 2026-06-21 07:51:04 on dist-test-slave-lzx3"
server_key: "3addffed38922cff6970490aeee4bf77"
server_key_iv: "ef23a738b1d9f92b41c9a1d9670accce"
server_key_version: "encryptionkey@0"
I20260621 07:51:04.473424 19928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "3ad41e6ee44d4216b55a1c823ea86682"
format_stamp: "Formatted at 2026-06-21 07:51:04 on dist-test-slave-lzx3"
server_key: "3addffed38922cff6970490aeee4bf77"
server_key_iv: "ef23a738b1d9f92b41c9a1d9670accce"
server_key_version: "encryptionkey@0"
I20260621 07:51:04.476390 19928 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260621 07:51:04.478466 19943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:04.479466 19928 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:51:04.479585 19928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "3ad41e6ee44d4216b55a1c823ea86682"
format_stamp: "Formatted at 2026-06-21 07:51:04 on dist-test-slave-lzx3"
server_key: "3addffed38922cff6970490aeee4bf77"
server_key_iv: "ef23a738b1d9f92b41c9a1d9670accce"
server_key_version: "encryptionkey@0"
I20260621 07:51:04.479683 19928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:04.491945 19928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:04.494537 19928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:04.494702 19928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:04.500424 19928 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:37995
I20260621 07:51:04.500478 19995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:37995 every 8 connection(s)
I20260621 07:51:04.501194 19928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:51:04.503453 19996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:04.507175 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 19928
I20260621 07:51:04.507313 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:51:04.507637 771 external_mini_cluster.cc:1428] Setting key 10f7d5c712b806d5435a6320c4ce955d
I20260621 07:51:04.508474 19996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ad41e6ee44d4216b55a1c823ea86682: Bootstrap starting.
I20260621 07:51:04.510852 19996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ad41e6ee44d4216b55a1c823ea86682: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:04.511576 19996 log.cc:824] T 00000000000000000000000000000000 P 3ad41e6ee44d4216b55a1c823ea86682: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:04.513206 19996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ad41e6ee44d4216b55a1c823ea86682: No bootstrap required, opened a new log
Jun 21 07:51:04 dist-test-slave-lzx3 krb5kdc[19915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:51:04.516855 19996 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ad41e6ee44d4216b55a1c823ea86682 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ad41e6ee44d4216b55a1c823ea86682" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 37995 } }
I20260621 07:51:04.517145 19996 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ad41e6ee44d4216b55a1c823ea86682 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:04.517205 19996 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ad41e6ee44d4216b55a1c823ea86682 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ad41e6ee44d4216b55a1c823ea86682, State: Initialized, Role: FOLLOWER
I20260621 07:51:04.517637 19996 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ad41e6ee44d4216b55a1c823ea86682 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ad41e6ee44d4216b55a1c823ea86682" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 37995 } }
I20260621 07:51:04.517797 19996 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ad41e6ee44d4216b55a1c823ea86682 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:51:04.517876 19996 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ad41e6ee44d4216b55a1c823ea86682 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:51:04.517966 19996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ad41e6ee44d4216b55a1c823ea86682 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:04.518903 19996 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ad41e6ee44d4216b55a1c823ea86682 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ad41e6ee44d4216b55a1c823ea86682" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 37995 } }
I20260621 07:51:04.519217 19996 leader_election.cc:302] T 00000000000000000000000000000000 P 3ad41e6ee44d4216b55a1c823ea86682 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ad41e6ee44d4216b55a1c823ea86682; no voters:
I20260621 07:51:04.519536 19996 leader_election.cc:288] T 00000000000000000000000000000000 P 3ad41e6ee44d4216b55a1c823ea86682 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:51:04.519694 20001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ad41e6ee44d4216b55a1c823ea86682 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:04.519922 20001 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ad41e6ee44d4216b55a1c823ea86682 [term 1 LEADER]: Becoming Leader. State: Replica: 3ad41e6ee44d4216b55a1c823ea86682, State: Running, Role: LEADER
I20260621 07:51:04.520190 20001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ad41e6ee44d4216b55a1c823ea86682 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ad41e6ee44d4216b55a1c823ea86682" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 37995 } }
I20260621 07:51:04.520582 19996 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ad41e6ee44d4216b55a1c823ea86682 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:51:04.520663 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:04.508855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49852 (local address 127.0.192.254:37995)
0621 07:51:04.509204 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:04.509219 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:04.509255 (+ 36us) server_negotiation.cc:407] Connection header received
0621 07:51:04.509985 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:04.510004 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:04.510347 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:04.510659 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:04.511328 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:04.512470 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:04.513245 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:04.513851 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:04.514008 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:04.516045 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:04.516053 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:04.516069 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:04.516095 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:04.517950 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:04.518357 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:04.518362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:04.518367 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:04.518493 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:04.518734 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:04.518738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:04.518740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:04.519172 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:04.519341 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:04.519625 (+ 284us) server_negotiation.cc:299] Negotiation successful
0621 07:51:04.519885 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":91,"threads_started":1}
I20260621 07:51:04.521250 20001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ad41e6ee44d4216b55a1c823ea86682 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ad41e6ee44d4216b55a1c823ea86682. Latest consensus state: current_term: 1 leader_uuid: "3ad41e6ee44d4216b55a1c823ea86682" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ad41e6ee44d4216b55a1c823ea86682" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 37995 } } }
I20260621 07:51:04.521364 20001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ad41e6ee44d4216b55a1c823ea86682 [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:04.521744 20010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:51:04.521713 20002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ad41e6ee44d4216b55a1c823ea86682 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ad41e6ee44d4216b55a1c823ea86682" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ad41e6ee44d4216b55a1c823ea86682" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 37995 } } }
I20260621 07:51:04.521881 20002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ad41e6ee44d4216b55a1c823ea86682 [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:04.524317 20010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:51:04.529215 20010 catalog_manager.cc:1347] Generated new cluster ID: 3d5e62f3bdf44181b08da08984ec03b9
I20260621 07:51:04.529273 20010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:51:04.547721 20010 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:51:04.548862 20010 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:51:04.554862 20010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ad41e6ee44d4216b55a1c823ea86682: Generated new TSK 0
I20260621 07:51:04.555532 20010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.192.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab.
Entry for principal kudu/127.0.192.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab.
I20260621 07:51:04.587098 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.193:0
--local_ip_for_outbound_sockets=127.0.192.193
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:37995
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:45179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:04.685693 20022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:04.685961 20022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:04.686028 20022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:04.689599 20022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:04.689662 20022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:04.689729 20022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.193
I20260621 07:51:04.694001 20022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:45179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/krb5kdc/kudu_127.0.192.193.keytab
--principal=kudu/127.0.192.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:37995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.193
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:04.694772 20022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:04.695595 20022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:04.703596 20028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:04.703572 20030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:04.703704 20022 server_base.cc:1034] running on GCE node
W20260621 07:51:04.703554 20027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:04.704048 20022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:04.704622 20022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:04.705842 20022 hybrid_clock.cc:648] HybridClock initialized: now 1782028264705832 us; error 30 us; skew 500 ppm
Jun 21 07:51:04 dist-test-slave-lzx3 krb5kdc[19915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028264, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:04.711223 20022 init.cc:377] Logged in from keytab as kudu/127.0.192.193@KRBTEST.COM (short username kudu)
I20260621 07:51:04.711917 20022 webserver.cc:460] Webserver started at http://127.0.192.193:33369/ using document root <none> and password file <none>
I20260621 07:51:04.712389 20022 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:04.712477 20022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:04.712707 20022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:04.714489 20022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-0/data/instance:
uuid: "ee45cbd449f04740bfedc60c08464749"
format_stamp: "Formatted at 2026-06-21 07:51:04 on dist-test-slave-lzx3"
server_key: "9f9973384f7aec7cd7af1f2127c5ca87"
server_key_iv: "e3599a617e47f572c78c9c1c90b78410"
server_key_version: "encryptionkey@0"
I20260621 07:51:04.715003 20022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-0/wal/instance:
uuid: "ee45cbd449f04740bfedc60c08464749"
format_stamp: "Formatted at 2026-06-21 07:51:04 on dist-test-slave-lzx3"
server_key: "9f9973384f7aec7cd7af1f2127c5ca87"
server_key_iv: "e3599a617e47f572c78c9c1c90b78410"
server_key_version: "encryptionkey@0"
I20260621 07:51:04.718317 20022 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 07:51:04.720232 20037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:04.721441 20022 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:51:04.721544 20022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-0/wal
uuid: "ee45cbd449f04740bfedc60c08464749"
format_stamp: "Formatted at 2026-06-21 07:51:04 on dist-test-slave-lzx3"
server_key: "9f9973384f7aec7cd7af1f2127c5ca87"
server_key_iv: "e3599a617e47f572c78c9c1c90b78410"
server_key_version: "encryptionkey@0"
I20260621 07:51:04.721632 20022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:04.750842 20022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:04.753628 20022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:04.753796 20022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:04.754282 20022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:51:04.754946 20022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:51:04.754987 20022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:04.755040 20022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:51:04.755069 20022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:04.762506 20022 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.193:41109
I20260621 07:51:04.762594 20150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.193:41109 every 8 connection(s)
I20260621 07:51:04.763293 20022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
Jun 21 07:51:04 dist-test-slave-lzx3 krb5kdc[19915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028264, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.193@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:51:04.771996 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 20022
I20260621 07:51:04.772110 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-0/wal/instance
I20260621 07:51:04.772430 771 external_mini_cluster.cc:1428] Setting key b5b359126550c656fd85350b0defe0ad
I20260621 07:51:04.775003 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:04.764818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:55927 (local address 127.0.192.254:37995)
0621 07:51:04.764931 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:04.764934 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:04.765595 (+ 661us) server_negotiation.cc:407] Connection header received
0621 07:51:04.766811 (+ 1216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:04.766814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:04.766869 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:04.766971 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:04.767995 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:04.768446 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:04.769432 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:04.769641 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:04.769739 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:04.771710 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:04.771712 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:04.771714 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:04.771733 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:04.773144 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:04.773607 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:04.773610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:04.773612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:04.773684 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:04.773996 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:04.774000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:04.774002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:04.774205 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:04.774294 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:04.774674 (+ 380us) server_negotiation.cc:299] Negotiation successful
0621 07:51:04.774792 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260621 07:51:04.775558 20153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:04.765112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:37995 (local address 127.0.192.193:55927)
0621 07:51:04.765475 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:04.765510 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:04.766614 (+ 1104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:04.767134 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:04.767141 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:04.767449 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:04.767854 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:04.767865 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:04.768590 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:04.768593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:04.769321 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:04.769329 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:04.769768 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:04.769772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:04.769910 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:04.770404 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:04.770417 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:04.771561 (+ 1144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:04.773247 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:04.773251 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:04.773263 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:04.773506 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:04.773779 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:04.773782 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:04.773783 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:04.773899 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:04.774293 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:04.774298 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:04.774476 (+ 178us) client_negotiation.cc:769] Sending connection context
0621 07:51:04.774607 (+ 131us) client_negotiation.cc:240] Negotiation successful
0621 07:51:04.774800 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":88,"threads_started":1}
I20260621 07:51:04.776638 20151 heartbeater.cc:347] Connected to a master server at 127.0.192.254:37995
I20260621 07:51:04.776847 20151 heartbeater.cc:464] Registering TS with master...
I20260621 07:51:04.777256 20151 heartbeater.cc:511] Master 127.0.192.254:37995 requested a full tablet report, sending...
I20260621 07:51:04.778601 19960 ts_manager.cc:194] Registered new tserver with Master: ee45cbd449f04740bfedc60c08464749 (127.0.192.193:41109)
I20260621 07:51:04.779531 19960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.193@KRBTEST.COM'} at 127.0.192.193:55927
WARNING: no policy specified for kudu/127.0.192.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab.
Entry for principal kudu/127.0.192.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab.
I20260621 07:51:04.795249 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.194:0
--local_ip_for_outbound_sockets=127.0.192.194
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:37995
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:45179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:04.890857 20156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:04.891067 20156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:04.891130 20156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:04.894485 20156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:04.894550 20156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:04.894619 20156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.194
I20260621 07:51:04.899047 20156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:45179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/krb5kdc/kudu_127.0.192.194.keytab
--principal=kudu/127.0.192.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:37995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.194
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:04.899801 20156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:04.900743 20156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:04.908759 20164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:04.908754 20161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:04.908949 20156 server_base.cc:1034] running on GCE node
W20260621 07:51:04.908759 20162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:04.909379 20156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:04.909901 20156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:04.911077 20156 hybrid_clock.cc:648] HybridClock initialized: now 1782028264911062 us; error 27 us; skew 500 ppm
Jun 21 07:51:04 dist-test-slave-lzx3 krb5kdc[19915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028264, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:04.913676 20156 init.cc:377] Logged in from keytab as kudu/127.0.192.194@KRBTEST.COM (short username kudu)
I20260621 07:51:04.914426 20156 webserver.cc:460] Webserver started at http://127.0.192.194:33375/ using document root <none> and password file <none>
I20260621 07:51:04.914886 20156 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:04.914973 20156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:04.915195 20156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:04.916988 20156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-1/data/instance:
uuid: "ec484099c1364c8ab62680920ff3fe3e"
format_stamp: "Formatted at 2026-06-21 07:51:04 on dist-test-slave-lzx3"
server_key: "894d1add5f58f6e7bd75687a114890b3"
server_key_iv: "2c2f65cec6a88b356127396e9522cd43"
server_key_version: "encryptionkey@0"
I20260621 07:51:04.917505 20156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-1/wal/instance:
uuid: "ec484099c1364c8ab62680920ff3fe3e"
format_stamp: "Formatted at 2026-06-21 07:51:04 on dist-test-slave-lzx3"
server_key: "894d1add5f58f6e7bd75687a114890b3"
server_key_iv: "2c2f65cec6a88b356127396e9522cd43"
server_key_version: "encryptionkey@0"
I20260621 07:51:04.920712 20156 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:51:04.922837 20171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:04.923816 20156 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:51:04.923933 20156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-1/wal
uuid: "ec484099c1364c8ab62680920ff3fe3e"
format_stamp: "Formatted at 2026-06-21 07:51:04 on dist-test-slave-lzx3"
server_key: "894d1add5f58f6e7bd75687a114890b3"
server_key_iv: "2c2f65cec6a88b356127396e9522cd43"
server_key_version: "encryptionkey@0"
I20260621 07:51:04.924024 20156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:04.938030 20156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:04.941146 20156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:04.941325 20156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:04.941828 20156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:51:04.942541 20156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:51:04.942605 20156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:04.942662 20156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:51:04.942703 20156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:04.950347 20156 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.194:38425
I20260621 07:51:04.950368 20284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.194:38425 every 8 connection(s)
I20260621 07:51:04.951056 20156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
Jun 21 07:51:04 dist-test-slave-lzx3 krb5kdc[19915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028264, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.194@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:51:04.960281 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 20156
I20260621 07:51:04.960412 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-1/wal/instance
I20260621 07:51:04.960706 771 external_mini_cluster.cc:1428] Setting key a36730f77572dccd975f42503b62ba99
I20260621 07:51:04.962260 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:04.952476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:38213 (local address 127.0.192.254:37995)
0621 07:51:04.952638 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:04.952642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:04.953297 (+ 655us) server_negotiation.cc:407] Connection header received
0621 07:51:04.954278 (+ 981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:04.954281 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:04.954321 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:04.954393 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:04.955293 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:04.955728 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:04.956719 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:04.956920 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:04.956995 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:04.958953 (+ 1958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:04.958956 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:04.958959 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:04.958978 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:04.960309 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:04.960867 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:04.960870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:04.960871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:04.960953 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:04.961284 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:04.961286 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:04.961287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:04.961488 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:04.961570 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:04.962023 (+ 453us) server_negotiation.cc:299] Negotiation successful
0621 07:51:04.962120 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 07:51:04.962864 20287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:04.952804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:37995 (local address 127.0.192.194:38213)
0621 07:51:04.953166 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:04.953200 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:04.954116 (+ 916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:04.954532 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:04.954538 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:04.954806 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:04.955186 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:04.955197 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:04.955815 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:04.955817 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:04.956611 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:04.956621 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:04.957036 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:04.957038 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:04.957163 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:04.957593 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:04.957608 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:04.958793 (+ 1185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:04.960435 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:04.960439 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:04.960468 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:04.960784 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:04.961050 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:04.961053 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:04.961054 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:04.961202 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:04.961577 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:04.961581 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:04.961792 (+ 211us) client_negotiation.cc:769] Sending connection context
0621 07:51:04.961970 (+ 178us) client_negotiation.cc:240] Negotiation successful
0621 07:51:04.962214 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":89,"threads_started":1}
I20260621 07:51:04.963847 20285 heartbeater.cc:347] Connected to a master server at 127.0.192.254:37995
I20260621 07:51:04.964061 20285 heartbeater.cc:464] Registering TS with master...
I20260621 07:51:04.964430 20285 heartbeater.cc:511] Master 127.0.192.254:37995 requested a full tablet report, sending...
I20260621 07:51:04.965269 19960 ts_manager.cc:194] Registered new tserver with Master: ec484099c1364c8ab62680920ff3fe3e (127.0.192.194:38425)
I20260621 07:51:04.965801 19960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.194@KRBTEST.COM'} at 127.0.192.194:38213
WARNING: no policy specified for kudu/127.0.192.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab.
Entry for principal kudu/127.0.192.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab.
I20260621 07:51:04.983085 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.195:0
--local_ip_for_outbound_sockets=127.0.192.195
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:37995
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:45179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:05.076570 20290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:05.076844 20290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:05.076884 20290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:05.080390 20290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:05.080438 20290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:05.080520 20290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.195
I20260621 07:51:05.085726 20290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:45179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/krb5kdc/kudu_127.0.192.195.keytab
--principal=kudu/127.0.192.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:37995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.195
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:05.086563 20290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:05.087404 20290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:05.095095 20298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:05.095057 20295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:05.095060 20296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:05.095261 20290 server_base.cc:1034] running on GCE node
I20260621 07:51:05.095695 20290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:05.096132 20290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:05.097275 20290 hybrid_clock.cc:648] HybridClock initialized: now 1782028265097253 us; error 33 us; skew 500 ppm
Jun 21 07:51:05 dist-test-slave-lzx3 krb5kdc[19915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028265, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:05.099555 20290 init.cc:377] Logged in from keytab as kudu/127.0.192.195@KRBTEST.COM (short username kudu)
I20260621 07:51:05.100183 20290 webserver.cc:460] Webserver started at http://127.0.192.195:44481/ using document root <none> and password file <none>
I20260621 07:51:05.100647 20290 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:05.100700 20290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:05.100873 20290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:05.102449 20290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-2/data/instance:
uuid: "e7a98d6e2a8a4105abfd2333825eb4b3"
format_stamp: "Formatted at 2026-06-21 07:51:05 on dist-test-slave-lzx3"
server_key: "e6d991fca1785ea2b6f4be67bd973cbd"
server_key_iv: "fb89faa956feaad349b866b87b2a1200"
server_key_version: "encryptionkey@0"
I20260621 07:51:05.102927 20290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-2/wal/instance:
uuid: "e7a98d6e2a8a4105abfd2333825eb4b3"
format_stamp: "Formatted at 2026-06-21 07:51:05 on dist-test-slave-lzx3"
server_key: "e6d991fca1785ea2b6f4be67bd973cbd"
server_key_iv: "fb89faa956feaad349b866b87b2a1200"
server_key_version: "encryptionkey@0"
I20260621 07:51:05.105960 20290 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260621 07:51:05.107992 20305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:05.109014 20290 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260621 07:51:05.109129 20290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-2/wal
uuid: "e7a98d6e2a8a4105abfd2333825eb4b3"
format_stamp: "Formatted at 2026-06-21 07:51:05 on dist-test-slave-lzx3"
server_key: "e6d991fca1785ea2b6f4be67bd973cbd"
server_key_iv: "fb89faa956feaad349b866b87b2a1200"
server_key_version: "encryptionkey@0"
I20260621 07:51:05.109220 20290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:05.127106 20290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:05.129849 20290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:05.130010 20290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:05.130457 20290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:51:05.131095 20290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:51:05.131157 20290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:05.131210 20290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:51:05.131250 20290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:05.139621 20290 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.195:40257
I20260621 07:51:05.139660 20418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.195:40257 every 8 connection(s)
I20260621 07:51:05.140362 20290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
I20260621 07:51:05.147437 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 20290
I20260621 07:51:05.147559 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-2/wal/instance
I20260621 07:51:05.147823 771 external_mini_cluster.cc:1428] Setting key ccf3bbd68b5274889cde944d97bd1697
Jun 21 07:51:05 dist-test-slave-lzx3 krb5kdc[19915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028265, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.195@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:51:05.152020 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:05.141804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:34413 (local address 127.0.192.254:37995)
0621 07:51:05.141921 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:05.141924 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:05.142571 (+ 647us) server_negotiation.cc:407] Connection header received
0621 07:51:05.143635 (+ 1064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:05.143638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:05.143691 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:05.143787 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:05.144728 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:05.145187 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:05.146293 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:05.146516 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:05.146627 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:05.149034 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:05.149041 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:05.149043 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:05.149083 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:05.150285 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:05.150702 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:05.150708 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:05.150709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:05.150786 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:05.151076 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:05.151080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:05.151082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:05.151236 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:05.151328 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:05.151750 (+ 422us) server_negotiation.cc:299] Negotiation successful
0621 07:51:05.151862 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260621 07:51:05.152735 20421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:05.142097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:37995 (local address 127.0.192.195:34413)
0621 07:51:05.142446 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:05.142480 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:05.143454 (+ 974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:05.143930 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:05.143936 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:05.144213 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:05.144601 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:05.144612 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:05.145318 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:05.145320 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:05.146164 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:05.146173 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:05.146657 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:05.146661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:05.146823 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:05.147372 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:05.147387 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:05.148885 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:05.150386 (+ 1501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:05.150391 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:05.150402 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:05.150611 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:05.150874 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:05.150876 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:05.150878 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:05.150994 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:05.151335 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:05.151343 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:05.151529 (+ 186us) client_negotiation.cc:769] Sending connection context
0621 07:51:05.151686 (+ 157us) client_negotiation.cc:240] Negotiation successful
0621 07:51:05.151984 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":89,"threads_started":1}
I20260621 07:51:05.153649 20419 heartbeater.cc:347] Connected to a master server at 127.0.192.254:37995
I20260621 07:51:05.153843 20419 heartbeater.cc:464] Registering TS with master...
I20260621 07:51:05.154210 20419 heartbeater.cc:511] Master 127.0.192.254:37995 requested a full tablet report, sending...
I20260621 07:51:05.154984 19960 ts_manager.cc:194] Registered new tserver with Master: e7a98d6e2a8a4105abfd2333825eb4b3 (127.0.192.195:40257)
I20260621 07:51:05.155450 19960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.192.195@KRBTEST.COM'} at 127.0.192.195:34413
I20260621 07:51:05.161422 771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:51:05.168977 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:05.162980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49856 (local address 127.0.192.254:37995)
0621 07:51:05.163088 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:05.163093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:05.163215 (+ 122us) server_negotiation.cc:407] Connection header received
0621 07:51:05.163291 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:05.163293 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:05.163319 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:05.163372 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:05.163898 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:05.164288 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:05.164964 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:05.165168 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:05.165250 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:05.166496 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:05.166499 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:05.166500 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:05.166516 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:05.167621 (+ 1105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:05.167979 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:05.167982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:05.167983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:05.168045 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:05.168254 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:05.168256 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:05.168257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:05.168422 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:05.168515 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:05.168704 (+ 189us) server_negotiation.cc:299] Negotiation successful
0621 07:51:05.168791 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260621 07:51:05.171229 19960 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:51:05.173803 19960 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:51:05.186262 20430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:05.180755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:38425 (local address 127.0.0.1:48438)
0621 07:51:05.181380 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:05.181424 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:05.181563 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:05.182179 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:05.182184 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:05.182211 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:51:05.182361 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:05.182369 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:05.183609 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:05.183611 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:05.184954 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:05.184963 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:05.185746 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:05.185749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:05.185852 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:05.185888 (+ 36us) client_negotiation.cc:769] Sending connection context
0621 07:51:05.186017 (+ 129us) client_negotiation.cc:240] Negotiation successful
0621 07:51:05.186129 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":134,"spinlock_wait_cycles":1664,"thread_start_us":48,"threads_started":1}
I20260621 07:51:05.186278 20431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:05.180887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48438 (local address 127.0.192.194:38425)
0621 07:51:05.181141 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:05.181152 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:05.181509 (+ 357us) server_negotiation.cc:407] Connection header received
0621 07:51:05.181704 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:05.181710 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:05.181898 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:05.182011 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:51:05.182504 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:05.183512 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:05.185095 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:05.185609 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:05.185738 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:05.185871 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:05.186045 (+ 174us) server_negotiation.cc:299] Negotiation successful
0621 07:51:05.186133 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":104,"threads_started":1}
I20260621 07:51:05.186684 20434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:05.181828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.195:40257 (local address 127.0.0.1:54420)
0621 07:51:05.182445 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:05.182457 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:05.182530 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:05.182952 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:05.182955 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:05.182972 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:51:05.183040 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:05.183044 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:05.184027 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:05.184031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:05.185001 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:05.185013 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:05.186126 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:05.186130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:05.186368 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:05.186424 (+ 56us) client_negotiation.cc:769] Sending connection context
0621 07:51:05.186496 (+ 72us) client_negotiation.cc:240] Negotiation successful
0621 07:51:05.186565 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"mutex_wait_us":122,"thread_start_us":71,"threads_started":1}
I20260621 07:51:05.186800 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:05.181213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37558 (local address 127.0.192.193:41109)
0621 07:51:05.181479 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:05.181489 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:05.181509 (+ 20us) server_negotiation.cc:407] Connection header received
0621 07:51:05.181722 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:05.181726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:05.181897 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:05.182018 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:51:05.182611 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:05.183427 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:05.185045 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:05.185844 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:05.185947 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:05.186062 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:05.186557 (+ 495us) server_negotiation.cc:299] Negotiation successful
0621 07:51:05.186646 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":109,"threads_started":1}
I20260621 07:51:05.187165 20435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:05.182136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54420 (local address 127.0.192.195:40257)
0621 07:51:05.182433 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:05.182442 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:05.182607 (+ 165us) server_negotiation.cc:407] Connection header received
0621 07:51:05.182663 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:05.182668 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:05.182826 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:05.182948 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:51:05.183155 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:05.183922 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:05.185120 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:05.185844 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:05.185947 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:05.186062 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:05.186961 (+ 899us) server_negotiation.cc:299] Negotiation successful
0621 07:51:05.187040 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":148,"threads_started":1}
I20260621 07:51:05.187156 20432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:05.180927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:41109 (local address 127.0.0.1:37558)
0621 07:51:05.181380 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:05.181424 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:05.181528 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:05.182179 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:05.182184 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:05.182211 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:51:05.182478 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:05.182486 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:05.183542 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:05.183548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:05.184923 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:05.184935 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:05.186126 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:05.186130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:05.186319 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:05.186422 (+ 103us) client_negotiation.cc:769] Sending connection context
0621 07:51:05.186969 (+ 547us) client_negotiation.cc:240] Negotiation successful
0621 07:51:05.187036 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":334,"spinlock_wait_cycles":16896,"thread_start_us":72,"threads_started":1}
I20260621 07:51:05.189272 20219 tablet_service.cc:1467] Processing CreateTablet for tablet c46b76c79025428f8e385f2212baca46 (DEFAULT_TABLE table=test-table [id=84cb6799b3324f129f91ce23dce16d64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:51:05.189587 20353 tablet_service.cc:1467] Processing CreateTablet for tablet c46b76c79025428f8e385f2212baca46 (DEFAULT_TABLE table=test-table [id=84cb6799b3324f129f91ce23dce16d64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:51:05.189656 20085 tablet_service.cc:1467] Processing CreateTablet for tablet c46b76c79025428f8e385f2212baca46 (DEFAULT_TABLE table=test-table [id=84cb6799b3324f129f91ce23dce16d64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:51:05.190238 20219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c46b76c79025428f8e385f2212baca46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:05.190299 20085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c46b76c79025428f8e385f2212baca46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:05.190348 20353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c46b76c79025428f8e385f2212baca46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:05.195267 20436 tablet_bootstrap.cc:492] T c46b76c79025428f8e385f2212baca46 P ee45cbd449f04740bfedc60c08464749: Bootstrap starting.
I20260621 07:51:05.197018 20437 tablet_bootstrap.cc:492] T c46b76c79025428f8e385f2212baca46 P e7a98d6e2a8a4105abfd2333825eb4b3: Bootstrap starting.
I20260621 07:51:05.197832 20436 tablet_bootstrap.cc:654] T c46b76c79025428f8e385f2212baca46 P ee45cbd449f04740bfedc60c08464749: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:05.198598 20437 tablet_bootstrap.cc:654] T c46b76c79025428f8e385f2212baca46 P e7a98d6e2a8a4105abfd2333825eb4b3: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:05.198771 20436 log.cc:824] T c46b76c79025428f8e385f2212baca46 P ee45cbd449f04740bfedc60c08464749: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:05.199303 20438 tablet_bootstrap.cc:492] T c46b76c79025428f8e385f2212baca46 P ec484099c1364c8ab62680920ff3fe3e: Bootstrap starting.
I20260621 07:51:05.199630 20437 log.cc:824] T c46b76c79025428f8e385f2212baca46 P e7a98d6e2a8a4105abfd2333825eb4b3: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:05.200644 20436 tablet_bootstrap.cc:492] T c46b76c79025428f8e385f2212baca46 P ee45cbd449f04740bfedc60c08464749: No bootstrap required, opened a new log
I20260621 07:51:05.200819 20436 ts_tablet_manager.cc:1397] T c46b76c79025428f8e385f2212baca46 P ee45cbd449f04740bfedc60c08464749: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260621 07:51:05.201265 20437 tablet_bootstrap.cc:492] T c46b76c79025428f8e385f2212baca46 P e7a98d6e2a8a4105abfd2333825eb4b3: No bootstrap required, opened a new log
I20260621 07:51:05.201469 20438 tablet_bootstrap.cc:654] T c46b76c79025428f8e385f2212baca46 P ec484099c1364c8ab62680920ff3fe3e: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:05.201546 20437 ts_tablet_manager.cc:1397] T c46b76c79025428f8e385f2212baca46 P e7a98d6e2a8a4105abfd2333825eb4b3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260621 07:51:05.202135 20438 log.cc:824] T c46b76c79025428f8e385f2212baca46 P ec484099c1364c8ab62680920ff3fe3e: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:05.203859 20438 tablet_bootstrap.cc:492] T c46b76c79025428f8e385f2212baca46 P ec484099c1364c8ab62680920ff3fe3e: No bootstrap required, opened a new log
I20260621 07:51:05.204026 20438 ts_tablet_manager.cc:1397] T c46b76c79025428f8e385f2212baca46 P ec484099c1364c8ab62680920ff3fe3e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260621 07:51:05.204428 20436 raft_consensus.cc:348] T c46b76c79025428f8e385f2212baca46 P ee45cbd449f04740bfedc60c08464749 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484099c1364c8ab62680920ff3fe3e" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 38425 } } peers { permanent_uuid: "ee45cbd449f04740bfedc60c08464749" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 41109 } } peers { permanent_uuid: "e7a98d6e2a8a4105abfd2333825eb4b3" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 40257 } }
I20260621 07:51:05.204689 20436 raft_consensus.cc:374] T c46b76c79025428f8e385f2212baca46 P ee45cbd449f04740bfedc60c08464749 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:05.204743 20436 raft_consensus.cc:729] T c46b76c79025428f8e385f2212baca46 P ee45cbd449f04740bfedc60c08464749 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee45cbd449f04740bfedc60c08464749, State: Initialized, Role: FOLLOWER
I20260621 07:51:05.204816 20437 raft_consensus.cc:348] T c46b76c79025428f8e385f2212baca46 P e7a98d6e2a8a4105abfd2333825eb4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484099c1364c8ab62680920ff3fe3e" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 38425 } } peers { permanent_uuid: "ee45cbd449f04740bfedc60c08464749" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 41109 } } peers { permanent_uuid: "e7a98d6e2a8a4105abfd2333825eb4b3" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 40257 } }
I20260621 07:51:05.205063 20437 raft_consensus.cc:374] T c46b76c79025428f8e385f2212baca46 P e7a98d6e2a8a4105abfd2333825eb4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:05.205121 20437 raft_consensus.cc:729] T c46b76c79025428f8e385f2212baca46 P e7a98d6e2a8a4105abfd2333825eb4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7a98d6e2a8a4105abfd2333825eb4b3, State: Initialized, Role: FOLLOWER
I20260621 07:51:05.205197 20436 consensus_queue.cc:260] T c46b76c79025428f8e385f2212baca46 P ee45cbd449f04740bfedc60c08464749 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484099c1364c8ab62680920ff3fe3e" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 38425 } } peers { permanent_uuid: "ee45cbd449f04740bfedc60c08464749" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 41109 } } peers { permanent_uuid: "e7a98d6e2a8a4105abfd2333825eb4b3" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 40257 } }
I20260621 07:51:05.205554 20437 consensus_queue.cc:260] T c46b76c79025428f8e385f2212baca46 P e7a98d6e2a8a4105abfd2333825eb4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484099c1364c8ab62680920ff3fe3e" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 38425 } } peers { permanent_uuid: "ee45cbd449f04740bfedc60c08464749" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 41109 } } peers { permanent_uuid: "e7a98d6e2a8a4105abfd2333825eb4b3" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 40257 } }
I20260621 07:51:05.205896 20151 heartbeater.cc:503] Master 127.0.192.254:37995 was elected leader, sending a full tablet report...
I20260621 07:51:05.206133 20436 ts_tablet_manager.cc:1428] T c46b76c79025428f8e385f2212baca46 P ee45cbd449f04740bfedc60c08464749: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260621 07:51:05.206419 20437 ts_tablet_manager.cc:1428] T c46b76c79025428f8e385f2212baca46 P e7a98d6e2a8a4105abfd2333825eb4b3: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260621 07:51:05.206718 20419 heartbeater.cc:503] Master 127.0.192.254:37995 was elected leader, sending a full tablet report...
I20260621 07:51:05.207518 20438 raft_consensus.cc:348] T c46b76c79025428f8e385f2212baca46 P ec484099c1364c8ab62680920ff3fe3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484099c1364c8ab62680920ff3fe3e" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 38425 } } peers { permanent_uuid: "ee45cbd449f04740bfedc60c08464749" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 41109 } } peers { permanent_uuid: "e7a98d6e2a8a4105abfd2333825eb4b3" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 40257 } }
I20260621 07:51:05.207721 20438 raft_consensus.cc:374] T c46b76c79025428f8e385f2212baca46 P ec484099c1364c8ab62680920ff3fe3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:05.207814 20438 raft_consensus.cc:729] T c46b76c79025428f8e385f2212baca46 P ec484099c1364c8ab62680920ff3fe3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec484099c1364c8ab62680920ff3fe3e, State: Initialized, Role: FOLLOWER
I20260621 07:51:05.208197 20438 consensus_queue.cc:260] T c46b76c79025428f8e385f2212baca46 P ec484099c1364c8ab62680920ff3fe3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484099c1364c8ab62680920ff3fe3e" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 38425 } } peers { permanent_uuid: "ee45cbd449f04740bfedc60c08464749" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 41109 } } peers { permanent_uuid: "e7a98d6e2a8a4105abfd2333825eb4b3" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 40257 } }
I20260621 07:51:05.208781 20285 heartbeater.cc:503] Master 127.0.192.254:37995 was elected leader, sending a full tablet report...
I20260621 07:51:05.209036 20438 ts_tablet_manager.cc:1428] T c46b76c79025428f8e385f2212baca46 P ec484099c1364c8ab62680920ff3fe3e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260621 07:51:05.264289 20152 tablet.cc:2369] T c46b76c79025428f8e385f2212baca46 P ee45cbd449f04740bfedc60c08464749: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 07:51:05.391252 20420 tablet.cc:2369] T c46b76c79025428f8e385f2212baca46 P e7a98d6e2a8a4105abfd2333825eb4b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 07:51:05.452076 20286 tablet.cc:2369] T c46b76c79025428f8e385f2212baca46 P ec484099c1364c8ab62680920ff3fe3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:51:05.490013 20442 raft_consensus.cc:482] T c46b76c79025428f8e385f2212baca46 P ee45cbd449f04740bfedc60c08464749 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:51:05.490183 20442 raft_consensus.cc:504] T c46b76c79025428f8e385f2212baca46 P ee45cbd449f04740bfedc60c08464749 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484099c1364c8ab62680920ff3fe3e" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 38425 } } peers { permanent_uuid: "ee45cbd449f04740bfedc60c08464749" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 41109 } } peers { permanent_uuid: "e7a98d6e2a8a4105abfd2333825eb4b3" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 40257 } }
I20260621 07:51:05.491248 20442 leader_election.cc:288] T c46b76c79025428f8e385f2212baca46 P ee45cbd449f04740bfedc60c08464749 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec484099c1364c8ab62680920ff3fe3e (127.0.192.194:38425), e7a98d6e2a8a4105abfd2333825eb4b3 (127.0.192.195:40257)
I20260621 07:51:05.494683 20445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:05.491393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:38425 (local address 127.0.192.193:33409)
0621 07:51:05.491656 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:05.491672 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:05.491801 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:05.492085 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:05.492090 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:05.492108 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:51:05.492235 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:05.492241 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:05.493003 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:05.493009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:05.493732 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:05.493740 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:05.494298 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:05.494301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:05.494384 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:05.494410 (+ 26us) client_negotiation.cc:769] Sending connection context
0621 07:51:05.494472 (+ 62us) client_negotiation.cc:240] Negotiation successful
0621 07:51:05.494536 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":65,"threads_started":1}
I20260621 07:51:05.494695 20431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:05.491487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:33409 (local address 127.0.192.194:38425)
0621 07:51:05.491636 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:05.491641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:05.491756 (+ 115us) server_negotiation.cc:407] Connection header received
0621 07:51:05.491920 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:05.491924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:05.491982 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:05.492046 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:51:05.492366 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:05.492898 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:05.493850 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:05.494193 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:05.494273 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:05.494344 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:05.494509 (+ 165us) server_negotiation.cc:299] Negotiation successful
0621 07:51:05.494586 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260621 07:51:05.494737 20435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:05.491676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:41041 (local address 127.0.192.195:40257)
0621 07:51:05.491810 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:05.491814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:05.491850 (+ 36us) server_negotiation.cc:407] Connection header received
0621 07:51:05.492002 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:05.492004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:05.492072 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:05.492152 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:51:05.492462 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:05.492965 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:05.493881 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:05.494215 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:05.494331 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:05.494378 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:05.494574 (+ 196us) server_negotiation.cc:299] Negotiation successful
0621 07:51:05.494635 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 07:51:05.494709 20446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:05.491485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.195:40257 (local address 127.0.192.193:41041)
0621 07:51:05.491767 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:05.491781 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:05.491904 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:05.492182 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:05.492184 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:05.492197 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:51:05.492320 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:05.492325 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:05.493074 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:05.493076 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:05.493747 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:05.493755 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:05.494345 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:05.494348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:05.494438 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:05.494470 (+ 32us) client_negotiation.cc:769] Sending connection context
0621 07:51:05.494540 (+ 70us) client_negotiation.cc:240] Negotiation successful
0621 07:51:05.494597 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":73,"threads_started":1}
I20260621 07:51:05.495325 20239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46b76c79025428f8e385f2212baca46" candidate_uuid: "ee45cbd449f04740bfedc60c08464749" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec484099c1364c8ab62680920ff3fe3e" is_pre_election: true
I20260621 07:51:05.495388 20373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46b76c79025428f8e385f2212baca46" candidate_uuid: "ee45cbd449f04740bfedc60c08464749" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7a98d6e2a8a4105abfd2333825eb4b3" is_pre_election: true
I20260621 07:51:05.495595 20239 raft_consensus.cc:2454] T c46b76c79025428f8e385f2212baca46 P ec484099c1364c8ab62680920ff3fe3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee45cbd449f04740bfedc60c08464749 in term 0.
I20260621 07:51:05.495649 20373 raft_consensus.cc:2454] T c46b76c79025428f8e385f2212baca46 P e7a98d6e2a8a4105abfd2333825eb4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee45cbd449f04740bfedc60c08464749 in term 0.
I20260621 07:51:05.496016 20041 leader_election.cc:302] T c46b76c79025428f8e385f2212baca46 P ee45cbd449f04740bfedc60c08464749 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ec484099c1364c8ab62680920ff3fe3e, ee45cbd449f04740bfedc60c08464749; no voters:
I20260621 07:51:05.496279 20442 raft_consensus.cc:2781] T c46b76c79025428f8e385f2212baca46 P ee45cbd449f04740bfedc60c08464749 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:51:05.496371 20442 raft_consensus.cc:482] T c46b76c79025428f8e385f2212baca46 P ee45cbd449f04740bfedc60c08464749 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:51:05.496428 20442 raft_consensus.cc:3037] T c46b76c79025428f8e385f2212baca46 P ee45cbd449f04740bfedc60c08464749 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:05.497440 20442 raft_consensus.cc:504] T c46b76c79025428f8e385f2212baca46 P ee45cbd449f04740bfedc60c08464749 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484099c1364c8ab62680920ff3fe3e" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 38425 } } peers { permanent_uuid: "ee45cbd449f04740bfedc60c08464749" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 41109 } } peers { permanent_uuid: "e7a98d6e2a8a4105abfd2333825eb4b3" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 40257 } }
I20260621 07:51:05.497709 20442 leader_election.cc:288] T c46b76c79025428f8e385f2212baca46 P ee45cbd449f04740bfedc60c08464749 [CANDIDATE]: Term 1 election: Requested vote from peers ec484099c1364c8ab62680920ff3fe3e (127.0.192.194:38425), e7a98d6e2a8a4105abfd2333825eb4b3 (127.0.192.195:40257)
I20260621 07:51:05.498006 20239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46b76c79025428f8e385f2212baca46" candidate_uuid: "ee45cbd449f04740bfedc60c08464749" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec484099c1364c8ab62680920ff3fe3e"
I20260621 07:51:05.498004 20373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46b76c79025428f8e385f2212baca46" candidate_uuid: "ee45cbd449f04740bfedc60c08464749" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7a98d6e2a8a4105abfd2333825eb4b3"
I20260621 07:51:05.498117 20373 raft_consensus.cc:3037] T c46b76c79025428f8e385f2212baca46 P e7a98d6e2a8a4105abfd2333825eb4b3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:05.498117 20239 raft_consensus.cc:3037] T c46b76c79025428f8e385f2212baca46 P ec484099c1364c8ab62680920ff3fe3e [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:05.499181 20239 raft_consensus.cc:2454] T c46b76c79025428f8e385f2212baca46 P ec484099c1364c8ab62680920ff3fe3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee45cbd449f04740bfedc60c08464749 in term 1.
I20260621 07:51:05.499181 20373 raft_consensus.cc:2454] T c46b76c79025428f8e385f2212baca46 P e7a98d6e2a8a4105abfd2333825eb4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee45cbd449f04740bfedc60c08464749 in term 1.
I20260621 07:51:05.499418 20041 leader_election.cc:302] T c46b76c79025428f8e385f2212baca46 P ee45cbd449f04740bfedc60c08464749 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ec484099c1364c8ab62680920ff3fe3e, ee45cbd449f04740bfedc60c08464749; no voters:
I20260621 07:51:05.499565 20442 raft_consensus.cc:2781] T c46b76c79025428f8e385f2212baca46 P ee45cbd449f04740bfedc60c08464749 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:05.499751 20442 raft_consensus.cc:686] T c46b76c79025428f8e385f2212baca46 P ee45cbd449f04740bfedc60c08464749 [term 1 LEADER]: Becoming Leader. State: Replica: ee45cbd449f04740bfedc60c08464749, State: Running, Role: LEADER
I20260621 07:51:05.500031 20442 consensus_queue.cc:237] T c46b76c79025428f8e385f2212baca46 P ee45cbd449f04740bfedc60c08464749 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484099c1364c8ab62680920ff3fe3e" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 38425 } } peers { permanent_uuid: "ee45cbd449f04740bfedc60c08464749" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 41109 } } peers { permanent_uuid: "e7a98d6e2a8a4105abfd2333825eb4b3" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 40257 } }
I20260621 07:51:05.502723 19958 catalog_manager.cc:5495] T c46b76c79025428f8e385f2212baca46 P ee45cbd449f04740bfedc60c08464749 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee45cbd449f04740bfedc60c08464749 (127.0.192.193). New cstate: current_term: 1 leader_uuid: "ee45cbd449f04740bfedc60c08464749" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484099c1364c8ab62680920ff3fe3e" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 38425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee45cbd449f04740bfedc60c08464749" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 41109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7a98d6e2a8a4105abfd2333825eb4b3" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 40257 } health_report { overall_health: UNKNOWN } } }
I20260621 07:51:05.542613 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:05.539146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37566 (local address 127.0.192.193:41109)
0621 07:51:05.539260 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:05.539264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:05.539277 (+ 13us) server_negotiation.cc:407] Connection header received
0621 07:51:05.539407 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:05.539410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:05.539460 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:05.539581 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:51:05.539827 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:05.540326 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:05.541168 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:05.541395 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:05.541522 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:05.541842 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:51:05.542209 (+ 367us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:51:05.542343 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:05.542409 (+ 66us) server_negotiation.cc:299] Negotiation successful
0621 07:51:05.542460 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260621 07:51:05.547539 20239 raft_consensus.cc:1261] T c46b76c79025428f8e385f2212baca46 P ec484099c1364c8ab62680920ff3fe3e [term 1 FOLLOWER]: Refusing update from remote peer ee45cbd449f04740bfedc60c08464749: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:51:05.547539 20373 raft_consensus.cc:1261] T c46b76c79025428f8e385f2212baca46 P e7a98d6e2a8a4105abfd2333825eb4b3 [term 1 FOLLOWER]: Refusing update from remote peer ee45cbd449f04740bfedc60c08464749: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:51:05.548125 20447 consensus_queue.cc:1035] T c46b76c79025428f8e385f2212baca46 P ee45cbd449f04740bfedc60c08464749 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec484099c1364c8ab62680920ff3fe3e" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 38425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:51:05.548251 20442 consensus_queue.cc:1035] T c46b76c79025428f8e385f2212baca46 P ee45cbd449f04740bfedc60c08464749 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7a98d6e2a8a4105abfd2333825eb4b3" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 40257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:51:05.553671 20450 mvcc.cc:205] Tried to move back new op lower bound from 7299187775677263872 to 7299187775491072000. Current Snapshot: MvccSnapshot[applied={T|T < 7299187775677263872}]
I20260621 07:51:05.554755 20452 mvcc.cc:205] Tried to move back new op lower bound from 7299187775677263872 to 7299187775491072000. Current Snapshot: MvccSnapshot[applied={T|T < 7299187775677263872}]
I20260621 07:51:05.562816 19960 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:51:05.562961 19960 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:51:05.564934 19960 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3ad41e6ee44d4216b55a1c823ea86682: Sending DeleteTablet for 3 replicas of tablet c46b76c79025428f8e385f2212baca46
I20260621 07:51:05.565639 20219 tablet_service.cc:1514] Processing DeleteTablet for tablet c46b76c79025428f8e385f2212baca46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:51:05 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:48438
I20260621 07:51:05.565714 20353 tablet_service.cc:1514] Processing DeleteTablet for tablet c46b76c79025428f8e385f2212baca46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:51:05 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:54420
I20260621 07:51:05.566120 20459 tablet_replica.cc:330] T c46b76c79025428f8e385f2212baca46 P ec484099c1364c8ab62680920ff3fe3e: stopping tablet replica
I20260621 07:51:05.566107 20085 tablet_service.cc:1514] Processing DeleteTablet for tablet c46b76c79025428f8e385f2212baca46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:51:05 UTC) from {username='kudu', principal='kudu/127.0.192.254@KRBTEST.COM'} at 127.0.0.1:37558
I20260621 07:51:05.566453 20459 raft_consensus.cc:2229] T c46b76c79025428f8e385f2212baca46 P ec484099c1364c8ab62680920ff3fe3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:51:05.566589 20461 tablet_replica.cc:330] T c46b76c79025428f8e385f2212baca46 P ee45cbd449f04740bfedc60c08464749: stopping tablet replica
I20260621 07:51:05.566758 20459 raft_consensus.cc:2258] T c46b76c79025428f8e385f2212baca46 P ec484099c1364c8ab62680920ff3fe3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:51:05.566934 20461 raft_consensus.cc:2229] T c46b76c79025428f8e385f2212baca46 P ee45cbd449f04740bfedc60c08464749 [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:51:05.567206 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 20022
I20260621 07:51:05.567337 20461 raft_consensus.cc:2258] T c46b76c79025428f8e385f2212baca46 P ee45cbd449f04740bfedc60c08464749 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:51:05.572894 20460 tablet_replica.cc:330] T c46b76c79025428f8e385f2212baca46 P e7a98d6e2a8a4105abfd2333825eb4b3: stopping tablet replica
I20260621 07:51:05.573153 20460 raft_consensus.cc:2229] T c46b76c79025428f8e385f2212baca46 P e7a98d6e2a8a4105abfd2333825eb4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:51:05.573411 20460 raft_consensus.cc:2258] T c46b76c79025428f8e385f2212baca46 P e7a98d6e2a8a4105abfd2333825eb4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:51:05.573720 20459 ts_tablet_manager.cc:1905] T c46b76c79025428f8e385f2212baca46 P ec484099c1364c8ab62680920ff3fe3e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260621 07:51:05.573932 19946 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.192.193:41109 (error 108)
I20260621 07:51:05.574802 20460 ts_tablet_manager.cc:1905] T c46b76c79025428f8e385f2212baca46 P e7a98d6e2a8a4105abfd2333825eb4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:51:05.574882 20432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:05.574590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:41109 (local address 127.0.0.1:37570)
0621 07:51:05.574711 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:05.574783 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.193:41109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260621 07:51:05.574941 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 20156
W20260621 07:51:05.576876 19946 catalog_manager.cc:4584] TS ee45cbd449f04740bfedc60c08464749 (127.0.192.193:41109): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c46b76c79025428f8e385f2212baca46: Network error: Client connection negotiation failed: client connection to 127.0.192.193:41109: connect: Connection refused (error 111)
I20260621 07:51:05.577203 20460 ts_tablet_manager.cc:1918] T c46b76c79025428f8e385f2212baca46 P e7a98d6e2a8a4105abfd2333825eb4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:51:05.577288 20460 log.cc:1196] T c46b76c79025428f8e385f2212baca46 P e7a98d6e2a8a4105abfd2333825eb4b3: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782028028522580-771-0/minicluster-data/ts-2/wal/wals/c46b76c79025428f8e385f2212baca46
I20260621 07:51:05.577613 20460 ts_tablet_manager.cc:1939] T c46b76c79025428f8e385f2212baca46 P e7a98d6e2a8a4105abfd2333825eb4b3: Deleting consensus metadata
I20260621 07:51:05.578540 19946 catalog_manager.cc:4841] TS e7a98d6e2a8a4105abfd2333825eb4b3 (127.0.192.195:40257): tablet c46b76c79025428f8e385f2212baca46 (table test-table [id=84cb6799b3324f129f91ce23dce16d64]) successfully deleted
I20260621 07:51:05.581511 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 20290
I20260621 07:51:05.582171 20432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:05.581622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:38425 (local address 127.0.0.1:48444)
0621 07:51:05.582067 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:05.582116 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.194:38425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380}
I20260621 07:51:05.586905 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 19928
2026-06-21T07:51:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2272 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-21T07:51:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:51:05Z Disabled control of system clock
I20260621 07:51:05.614507 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:39841
--webserver_interface=127.0.192.254
--webserver_port=0
--builtin_ntp_servers=127.0.192.212:37865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:39841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 07:51:05.710141 20466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:05.710350 20466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:05.710381 20466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:05.713876 20466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:51:05.713919 20466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:05.713933 20466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:05.713948 20466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:51:05.713960 20466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:51:05.718609 20466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:37865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:39841
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:39841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:05.719511 20466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:05.720676 20466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:05.728010 20471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:05.728040 20474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:05.728010 20472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:05.728080 20466 server_base.cc:1034] running on GCE node
I20260621 07:51:05.728750 20466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:05.729684 20466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:05.730859 20466 hybrid_clock.cc:648] HybridClock initialized: now 1782028265730838 us; error 41 us; skew 500 ppm
I20260621 07:51:05.732405 20466 webserver.cc:460] Webserver started at http://127.0.192.254:44781/ using document root <none> and password file <none>
I20260621 07:51:05.732993 20466 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:05.733106 20466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:05.733336 20466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:05.735242 20466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "54f1b422929d418095ad5010754121e8"
format_stamp: "Formatted at 2026-06-21 07:51:05 on dist-test-slave-lzx3"
server_key: "d1286058e1b3d6f71bd6cd2ff0078c7a"
server_key_iv: "72c85e346f7fb1178097ac53119053af"
server_key_version: "encryptionkey@0"
I20260621 07:51:05.735786 20466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "54f1b422929d418095ad5010754121e8"
format_stamp: "Formatted at 2026-06-21 07:51:05 on dist-test-slave-lzx3"
server_key: "d1286058e1b3d6f71bd6cd2ff0078c7a"
server_key_iv: "72c85e346f7fb1178097ac53119053af"
server_key_version: "encryptionkey@0"
I20260621 07:51:05.739239 20466 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260621 07:51:05.741389 20480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:05.742430 20466 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260621 07:51:05.742561 20466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "54f1b422929d418095ad5010754121e8"
format_stamp: "Formatted at 2026-06-21 07:51:05 on dist-test-slave-lzx3"
server_key: "d1286058e1b3d6f71bd6cd2ff0078c7a"
server_key_iv: "72c85e346f7fb1178097ac53119053af"
server_key_version: "encryptionkey@0"
I20260621 07:51:05.742659 20466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:05.757874 20466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:05.758435 20466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:05.758630 20466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:05.764642 20532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:39841 every 8 connection(s)
I20260621 07:51:05.764657 20466 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:39841
I20260621 07:51:05.765492 20466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:51:05.768108 20533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:05.768985 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 20466
I20260621 07:51:05.769109 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:51:05.769379 771 external_mini_cluster.cc:1428] Setting key fb024a72cb99fcdd31fce705da2da650
I20260621 07:51:05.773906 20533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54f1b422929d418095ad5010754121e8: Bootstrap starting.
I20260621 07:51:05.775982 20533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54f1b422929d418095ad5010754121e8: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:05.776788 20533 log.cc:824] T 00000000000000000000000000000000 P 54f1b422929d418095ad5010754121e8: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:05.778221 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:05.770656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52500 (local address 127.0.192.254:39841)
0621 07:51:05.771248 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:05.771266 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:05.771303 (+ 37us) server_negotiation.cc:407] Connection header received
0621 07:51:05.772037 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:05.772056 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:05.772492 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:05.772825 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:05.773217 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:05.774385 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:05.775129 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:05.775671 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:05.775830 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:05.776605 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:05.776611 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:05.776625 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:51:05.777099 (+ 474us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:05.777164 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:51:05.777184 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:05.777293 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:05.777445 (+ 152us) server_negotiation.cc:299] Negotiation successful
0621 07:51:05.777654 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":101,"threads_started":1}
I20260621 07:51:05.778554 20533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54f1b422929d418095ad5010754121e8: No bootstrap required, opened a new log
I20260621 07:51:05.781781 20533 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54f1b422929d418095ad5010754121e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54f1b422929d418095ad5010754121e8" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 39841 } }
I20260621 07:51:05.781993 20533 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54f1b422929d418095ad5010754121e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:05.782064 20533 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54f1b422929d418095ad5010754121e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54f1b422929d418095ad5010754121e8, State: Initialized, Role: FOLLOWER
I20260621 07:51:05.782444 20533 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54f1b422929d418095ad5010754121e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54f1b422929d418095ad5010754121e8" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 39841 } }
I20260621 07:51:05.782584 20533 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54f1b422929d418095ad5010754121e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:51:05.782665 20533 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54f1b422929d418095ad5010754121e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:51:05.782779 20533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54f1b422929d418095ad5010754121e8 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:05.783560 20533 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54f1b422929d418095ad5010754121e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54f1b422929d418095ad5010754121e8" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 39841 } }
I20260621 07:51:05.783885 20533 leader_election.cc:302] T 00000000000000000000000000000000 P 54f1b422929d418095ad5010754121e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54f1b422929d418095ad5010754121e8; no voters:
I20260621 07:51:05.784183 20533 leader_election.cc:288] T 00000000000000000000000000000000 P 54f1b422929d418095ad5010754121e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:51:05.784277 20538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54f1b422929d418095ad5010754121e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:05.784507 20538 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54f1b422929d418095ad5010754121e8 [term 1 LEADER]: Becoming Leader. State: Replica: 54f1b422929d418095ad5010754121e8, State: Running, Role: LEADER
I20260621 07:51:05.784811 20538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54f1b422929d418095ad5010754121e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54f1b422929d418095ad5010754121e8" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 39841 } }
I20260621 07:51:05.785342 20533 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54f1b422929d418095ad5010754121e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:51:05.786275 20540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54f1b422929d418095ad5010754121e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54f1b422929d418095ad5010754121e8. Latest consensus state: current_term: 1 leader_uuid: "54f1b422929d418095ad5010754121e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54f1b422929d418095ad5010754121e8" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 39841 } } }
I20260621 07:51:05.786412 20540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54f1b422929d418095ad5010754121e8 [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:05.786527 20539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54f1b422929d418095ad5010754121e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54f1b422929d418095ad5010754121e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54f1b422929d418095ad5010754121e8" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 39841 } } }
I20260621 07:51:05.786679 20539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54f1b422929d418095ad5010754121e8 [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:05.786834 20547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:51:05.789671 20547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:51:05.794385 20547 catalog_manager.cc:1347] Generated new cluster ID: 32cf13de4be54ccdaee2ad04136e2b71
I20260621 07:51:05.794461 20547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:51:05.804141 20547 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:51:05.805197 20547 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:51:05.815227 20547 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54f1b422929d418095ad5010754121e8: Generated new TSK 0
I20260621 07:51:05.815842 20547 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:51:05.817960 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.193:0
--local_ip_for_outbound_sockets=127.0.192.193
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:39841
--builtin_ntp_servers=127.0.192.212:37865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 07:51:05.917920 20557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:05.918119 20557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:05.918149 20557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:05.921527 20557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:05.921566 20557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:05.921609 20557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.193
I20260621 07:51:05.925786 20557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:37865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.192.193
--webserver_port=0
--tserver_master_addrs=127.0.192.254:39841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.193
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:05.926571 20557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:05.927644 20557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:05.939244 20563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:05.939260 20565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:05.939661 20562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:05.939666 20557 server_base.cc:1034] running on GCE node
I20260621 07:51:05.940258 20557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:05.941089 20557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:05.942267 20557 hybrid_clock.cc:648] HybridClock initialized: now 1782028265942243 us; error 40 us; skew 500 ppm
I20260621 07:51:05.944114 20557 webserver.cc:460] Webserver started at http://127.0.192.193:38861/ using document root <none> and password file <none>
I20260621 07:51:05.944810 20557 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:05.944892 20557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:05.945134 20557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:05.947913 20557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-0/data/instance:
uuid: "8a23f72ce2f14945ac5cfdfab3bbbec7"
format_stamp: "Formatted at 2026-06-21 07:51:05 on dist-test-slave-lzx3"
server_key: "78b89d4415a8b6ed3e31e9bc7d9cf4f3"
server_key_iv: "390b103f5e243b04cf648e046088b5ae"
server_key_version: "encryptionkey@0"
I20260621 07:51:05.948585 20557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-0/wal/instance:
uuid: "8a23f72ce2f14945ac5cfdfab3bbbec7"
format_stamp: "Formatted at 2026-06-21 07:51:05 on dist-test-slave-lzx3"
server_key: "78b89d4415a8b6ed3e31e9bc7d9cf4f3"
server_key_iv: "390b103f5e243b04cf648e046088b5ae"
server_key_version: "encryptionkey@0"
I20260621 07:51:05.952929 20557 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260621 07:51:05.955525 20571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:05.957064 20557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:51:05.957190 20557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-0/wal
uuid: "8a23f72ce2f14945ac5cfdfab3bbbec7"
format_stamp: "Formatted at 2026-06-21 07:51:05 on dist-test-slave-lzx3"
server_key: "78b89d4415a8b6ed3e31e9bc7d9cf4f3"
server_key_iv: "390b103f5e243b04cf648e046088b5ae"
server_key_version: "encryptionkey@0"
I20260621 07:51:05.957325 20557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:05.979944 20557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:05.980445 20557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:05.980621 20557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:05.981108 20557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:51:05.981896 20557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:51:05.981945 20557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:05.982014 20557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:51:05.982041 20557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:05.989786 20557 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.193:35577
I20260621 07:51:05.989830 20684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.193:35577 every 8 connection(s)
I20260621 07:51:05.990491 20557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-0/data/info.pb
I20260621 07:51:05.994179 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 20557
I20260621 07:51:05.994304 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-0/wal/instance
I20260621 07:51:05.994573 771 external_mini_cluster.cc:1428] Setting key 5292b76e3f829cc7141bc39657b6ded9
I20260621 07:51:05.996286 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.194:0
--local_ip_for_outbound_sockets=127.0.192.194
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:39841
--builtin_ntp_servers=127.0.192.212:37865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 07:51:05.998880 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:05.991916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:52797 (local address 127.0.192.254:39841)
0621 07:51:05.992034 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:05.992037 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:05.992898 (+ 861us) server_negotiation.cc:407] Connection header received
0621 07:51:05.994052 (+ 1154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:05.994055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:05.994096 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:05.994161 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:05.995107 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:05.995652 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:05.996905 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:05.997329 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:05.997438 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:05.998480 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:05.998483 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:05.998485 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:51:05.998496 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:05.998512 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:51:05.998519 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:05.998574 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:05.998740 (+ 166us) server_negotiation.cc:299] Negotiation successful
0621 07:51:05.998783 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260621 07:51:05.999789 20687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:05.992211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:39841 (local address 127.0.192.193:52797)
0621 07:51:05.992779 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:05.992811 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:05.993845 (+ 1034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:05.994366 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:05.994373 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:05.994472 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:05.995009 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:05.995022 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:05.995769 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:05.995772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:05.996787 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:05.996796 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:05.997460 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:05.997462 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:05.997679 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:05.998209 (+ 530us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:51:05.998211 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:05.998259 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:51:05.998332 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:05.998598 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:05.998608 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:05.998613 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 07:51:05.998736 (+ 123us) client_negotiation.cc:240] Negotiation successful
0621 07:51:05.998969 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":104,"threads_started":1}
I20260621 07:51:06.000875 20685 heartbeater.cc:347] Connected to a master server at 127.0.192.254:39841
I20260621 07:51:06.001063 20685 heartbeater.cc:464] Registering TS with master...
I20260621 07:51:06.001462 20685 heartbeater.cc:511] Master 127.0.192.254:39841 requested a full tablet report, sending...
I20260621 07:51:06.002842 20497 ts_manager.cc:194] Registered new tserver with Master: 8a23f72ce2f14945ac5cfdfab3bbbec7 (127.0.192.193:35577)
I20260621 07:51:06.004321 20497 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.192.193:52797
W20260621 07:51:06.092434 20688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:06.092674 20688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:06.092710 20688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:06.095911 20688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:06.095944 20688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:06.095988 20688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.194
I20260621 07:51:06.099974 20688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:37865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.192.194
--webserver_port=0
--tserver_master_addrs=127.0.192.254:39841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.194
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:06.100734 20688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:06.101797 20688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:06.110095 20696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:06.110178 20693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:06.110257 20688 server_base.cc:1034] running on GCE node
W20260621 07:51:06.110095 20694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:06.110743 20688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:06.111246 20688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:06.112406 20688 hybrid_clock.cc:648] HybridClock initialized: now 1782028266112384 us; error 37 us; skew 500 ppm
I20260621 07:51:06.113850 20688 webserver.cc:460] Webserver started at http://127.0.192.194:33849/ using document root <none> and password file <none>
I20260621 07:51:06.114293 20688 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:06.114346 20688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:06.114521 20688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:06.116175 20688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-1/data/instance:
uuid: "aa036f3224ca4c209ca0ee35a6b59fc6"
format_stamp: "Formatted at 2026-06-21 07:51:06 on dist-test-slave-lzx3"
server_key: "fcf06df31aa5be3dda244918ba420777"
server_key_iv: "7a2f3ac8706f9215dfadef75fe5b6259"
server_key_version: "encryptionkey@0"
I20260621 07:51:06.116732 20688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-1/wal/instance:
uuid: "aa036f3224ca4c209ca0ee35a6b59fc6"
format_stamp: "Formatted at 2026-06-21 07:51:06 on dist-test-slave-lzx3"
server_key: "fcf06df31aa5be3dda244918ba420777"
server_key_iv: "7a2f3ac8706f9215dfadef75fe5b6259"
server_key_version: "encryptionkey@0"
I20260621 07:51:06.119724 20688 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260621 07:51:06.121819 20702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:06.122786 20688 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 07:51:06.122895 20688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-1/wal
uuid: "aa036f3224ca4c209ca0ee35a6b59fc6"
format_stamp: "Formatted at 2026-06-21 07:51:06 on dist-test-slave-lzx3"
server_key: "fcf06df31aa5be3dda244918ba420777"
server_key_iv: "7a2f3ac8706f9215dfadef75fe5b6259"
server_key_version: "encryptionkey@0"
I20260621 07:51:06.122978 20688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:06.134091 20688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:06.134621 20688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:06.134809 20688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:06.135304 20688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:51:06.136162 20688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:51:06.136207 20688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:06.136272 20688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:51:06.136305 20688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:06.144311 20688 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.194:32927
I20260621 07:51:06.144351 20815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.194:32927 every 8 connection(s)
I20260621 07:51:06.145112 20688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-1/data/info.pb
I20260621 07:51:06.151896 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 20688
I20260621 07:51:06.152016 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-1/wal/instance
I20260621 07:51:06.152314 771 external_mini_cluster.cc:1428] Setting key d6da47d9308f9417f00e633290682d5d
I20260621 07:51:06.153064 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:06.146771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.194:56027 (local address 127.0.192.254:39841)
0621 07:51:06.146895 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:06.146898 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:06.147814 (+ 916us) server_negotiation.cc:407] Connection header received
0621 07:51:06.149066 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:06.149069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:06.149120 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:06.149212 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:06.149910 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:06.150369 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:06.151314 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:06.151505 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:06.151604 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:06.152603 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:06.152607 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:06.152610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:51:06.152626 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:06.152645 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:51:06.152653 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:06.152736 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:06.152882 (+ 146us) server_negotiation.cc:299] Negotiation successful
0621 07:51:06.152927 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 07:51:06.153842 20818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:06.147131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:39841 (local address 127.0.192.194:56027)
0621 07:51:06.147696 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:06.147727 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:06.148893 (+ 1166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:06.149350 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:06.149355 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:06.149422 (+ 67us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:06.149793 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:06.149802 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:06.150478 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:06.150482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:06.151201 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:06.151209 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:06.151628 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:06.151632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:06.151823 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:06.152331 (+ 508us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:51:06.152332 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:06.152378 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:51:06.152441 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:06.152748 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:06.152758 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:06.152763 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 07:51:06.152877 (+ 114us) client_negotiation.cc:240] Negotiation successful
0621 07:51:06.153095 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":115,"threads_started":1}
I20260621 07:51:06.154053 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.192.195:0
--local_ip_for_outbound_sockets=127.0.192.195
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:39841
--builtin_ntp_servers=127.0.192.212:37865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 07:51:06.154767 20816 heartbeater.cc:347] Connected to a master server at 127.0.192.254:39841
I20260621 07:51:06.154965 20816 heartbeater.cc:464] Registering TS with master...
I20260621 07:51:06.155360 20816 heartbeater.cc:511] Master 127.0.192.254:39841 requested a full tablet report, sending...
I20260621 07:51:06.156251 20497 ts_manager.cc:194] Registered new tserver with Master: aa036f3224ca4c209ca0ee35a6b59fc6 (127.0.192.194:32927)
I20260621 07:51:06.156865 20497 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.192.194:56027
W20260621 07:51:06.249432 20819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:06.249650 20819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:06.249711 20819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:06.253168 20819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:06.253228 20819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:06.253305 20819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.192.195
I20260621 07:51:06.257346 20819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:37865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.192.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.192.195
--webserver_port=0
--tserver_master_addrs=127.0.192.254:39841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.192.195
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:06.258117 20819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:06.259268 20819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 07:51:06.267313 20819 server_base.cc:1034] running on GCE node
W20260621 07:51:06.267272 20824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:06.267266 20827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:06.267272 20825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:06.267714 20819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:06.268229 20819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:06.269383 20819 hybrid_clock.cc:648] HybridClock initialized: now 1782028266269358 us; error 26 us; skew 500 ppm
I20260621 07:51:06.270835 20819 webserver.cc:460] Webserver started at http://127.0.192.195:37181/ using document root <none> and password file <none>
I20260621 07:51:06.271301 20819 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:06.271349 20819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:06.271574 20819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:06.273314 20819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-2/data/instance:
uuid: "80029e6fdf4b40a5bea1445789d65b1b"
format_stamp: "Formatted at 2026-06-21 07:51:06 on dist-test-slave-lzx3"
server_key: "1c6828238efd4ebe056dd94874321a15"
server_key_iv: "b73732e414c78f5d926392f765d7dca7"
server_key_version: "encryptionkey@0"
I20260621 07:51:06.273769 20819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-2/wal/instance:
uuid: "80029e6fdf4b40a5bea1445789d65b1b"
format_stamp: "Formatted at 2026-06-21 07:51:06 on dist-test-slave-lzx3"
server_key: "1c6828238efd4ebe056dd94874321a15"
server_key_iv: "b73732e414c78f5d926392f765d7dca7"
server_key_version: "encryptionkey@0"
I20260621 07:51:06.276894 20819 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260621 07:51:06.279120 20833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:06.280158 20819 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:51:06.280280 20819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-2/wal
uuid: "80029e6fdf4b40a5bea1445789d65b1b"
format_stamp: "Formatted at 2026-06-21 07:51:06 on dist-test-slave-lzx3"
server_key: "1c6828238efd4ebe056dd94874321a15"
server_key_iv: "b73732e414c78f5d926392f765d7dca7"
server_key_version: "encryptionkey@0"
I20260621 07:51:06.280375 20819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:06.290088 20819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:06.290756 20819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:06.290905 20819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:06.291553 20819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:51:06.292754 20819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:51:06.292817 20819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:06.292879 20819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:51:06.292901 20819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:06.301770 20819 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.195:46279
I20260621 07:51:06.301793 20946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.195:46279 every 8 connection(s)
I20260621 07:51:06.302546 20819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-2/data/info.pb
I20260621 07:51:06.309235 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 20819
I20260621 07:51:06.309372 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-2/wal/instance
I20260621 07:51:06.309659 771 external_mini_cluster.cc:1428] Setting key 36420209a4d764942f47f3625e18303f
I20260621 07:51:06.311100 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:06.304080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.195:40229 (local address 127.0.192.254:39841)
0621 07:51:06.304206 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:06.304211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:06.305249 (+ 1038us) server_negotiation.cc:407] Connection header received
0621 07:51:06.306551 (+ 1302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:06.306553 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:06.306598 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:06.306664 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:06.307530 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:06.308017 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:06.309079 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:06.309429 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:06.309541 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:06.310633 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:06.310637 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:06.310639 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:51:06.310651 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:06.310671 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:51:06.310680 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:06.310761 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:06.310926 (+ 165us) server_negotiation.cc:299] Negotiation successful
0621 07:51:06.310984 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260621 07:51:06.311954 20949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:06.304467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:39841 (local address 127.0.192.195:40229)
0621 07:51:06.305116 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:06.305152 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:06.306307 (+ 1155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:06.306852 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:06.306866 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:06.306984 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:06.307413 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:06.307424 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:06.308118 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:06.308120 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:06.308940 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:06.308950 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:06.309566 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:06.309569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:06.309741 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:06.310339 (+ 598us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:51:06.310341 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:06.310378 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:51:06.310432 (+ 54us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:06.310776 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:06.310785 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:06.310791 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 07:51:06.310912 (+ 121us) client_negotiation.cc:240] Negotiation successful
0621 07:51:06.311157 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":153,"threads_started":1}
I20260621 07:51:06.312908 20947 heartbeater.cc:347] Connected to a master server at 127.0.192.254:39841
I20260621 07:51:06.313103 20947 heartbeater.cc:464] Registering TS with master...
I20260621 07:51:06.313534 20947 heartbeater.cc:511] Master 127.0.192.254:39841 requested a full tablet report, sending...
I20260621 07:51:06.314414 20497 ts_manager.cc:194] Registered new tserver with Master: 80029e6fdf4b40a5bea1445789d65b1b (127.0.192.195:46279)
I20260621 07:51:06.314891 20497 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.192.195:40229
I20260621 07:51:06.323148 771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:51:06.328356 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:06.324948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52510 (local address 127.0.192.254:39841)
0621 07:51:06.325080 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:06.325086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:06.325268 (+ 182us) server_negotiation.cc:407] Connection header received
0621 07:51:06.325371 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:06.325373 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:06.325407 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:06.325471 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:06.325701 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:06.326209 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:06.327030 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:06.327239 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:06.327303 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:06.327995 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:06.327998 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:06.328000 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:51:06.328010 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:06.328023 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:51:06.328029 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:06.328079 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:06.328140 (+ 61us) server_negotiation.cc:299] Negotiation successful
0621 07:51:06.328188 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 07:51:06.330719 20497 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:51:06.333469 20497 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:51:06.347332 20959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:06.342225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.195:46279 (local address 127.0.0.1:45882)
0621 07:51:06.342765 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:06.342777 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:06.342850 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:06.343283 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:06.343289 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:06.343341 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:51:06.343575 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:06.343585 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:06.344610 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:06.344614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:06.345749 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:06.345759 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:06.346816 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:06.346820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:06.346929 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:06.346966 (+ 37us) client_negotiation.cc:769] Sending connection context
0621 07:51:06.347071 (+ 105us) client_negotiation.cc:240] Negotiation successful
0621 07:51:06.347189 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"mutex_wait_us":56,"spinlock_wait_cycles":3840,"thread_start_us":116,"threads_started":1}
I20260621 07:51:06.347386 20962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:06.342419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45882 (local address 127.0.192.195:46279)
0621 07:51:06.342811 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:06.342822 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:06.342858 (+ 36us) server_negotiation.cc:407] Connection header received
0621 07:51:06.342932 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:06.342937 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:06.343123 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:06.343445 (+ 322us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:51:06.343687 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:06.344469 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:06.345932 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:06.346696 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:06.346803 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:06.347053 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:06.347106 (+ 53us) server_negotiation.cc:299] Negotiation successful
0621 07:51:06.347230 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":113,"threads_started":1}
I20260621 07:51:06.347666 20958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:06.341546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:32927 (local address 127.0.0.1:50630)
0621 07:51:06.341858 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:06.341902 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:06.342016 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:06.343603 (+ 1587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:06.343607 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:06.343626 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:51:06.343730 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:06.343737 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:06.345258 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:06.345262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:06.346224 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:06.346232 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:06.347258 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:06.347262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:06.347365 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:06.347393 (+ 28us) client_negotiation.cc:769] Sending connection context
0621 07:51:06.347447 (+ 54us) client_negotiation.cc:240] Negotiation successful
0621 07:51:06.347522 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":24,"thread_start_us":85,"threads_started":1}
I20260621 07:51:06.348203 20963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:06.342540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50630 (local address 127.0.192.194:32927)
0621 07:51:06.343109 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:06.343119 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:06.343139 (+ 20us) server_negotiation.cc:407] Connection header received
0621 07:51:06.343219 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:06.343223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:06.343440 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:06.343628 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:51:06.343845 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:06.345094 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:06.346335 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:06.347129 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:06.347839 (+ 710us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:06.347995 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:06.348032 (+ 37us) server_negotiation.cc:299] Negotiation successful
0621 07:51:06.348091 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":81,"threads_started":1}
I20260621 07:51:06.348719 20961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:06.341592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:35577 (local address 127.0.0.1:32996)
0621 07:51:06.342466 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:06.342479 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:06.342579 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:06.343283 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:06.343289 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:06.343349 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:51:06.343513 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:06.343521 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:06.345248 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:06.345252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:06.347112 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:06.347120 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:06.348289 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:06.348293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:06.348379 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:06.348398 (+ 19us) client_negotiation.cc:769] Sending connection context
0621 07:51:06.348501 (+ 103us) client_negotiation.cc:240] Negotiation successful
0621 07:51:06.348587 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"mutex_wait_us":31,"spinlock_wait_cycles":1280,"thread_start_us":104,"threads_started":1}
I20260621 07:51:06.349431 20960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:06.341967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32996 (local address 127.0.192.193:35577)
0621 07:51:06.342408 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:06.342420 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:06.342700 (+ 280us) server_negotiation.cc:407] Connection header received
0621 07:51:06.342786 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:06.342794 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:06.343003 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:06.343147 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:51:06.343837 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:06.345093 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:06.347235 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:06.348159 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:06.348268 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:06.349118 (+ 850us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:06.349166 (+ 48us) server_negotiation.cc:299] Negotiation successful
0621 07:51:06.349243 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":117,"threads_started":1}
I20260621 07:51:06.350324 20881 tablet_service.cc:1467] Processing CreateTablet for tablet 6d0214cc69f1481d8a44bd1064084aed (DEFAULT_TABLE table=test-table [id=992787d620c84c0babb808ec33766e25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:51:06.350657 20750 tablet_service.cc:1467] Processing CreateTablet for tablet 6d0214cc69f1481d8a44bd1064084aed (DEFAULT_TABLE table=test-table [id=992787d620c84c0babb808ec33766e25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:51:06.351346 20750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d0214cc69f1481d8a44bd1064084aed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:06.351346 20881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d0214cc69f1481d8a44bd1064084aed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:06.351835 20619 tablet_service.cc:1467] Processing CreateTablet for tablet 6d0214cc69f1481d8a44bd1064084aed (DEFAULT_TABLE table=test-table [id=992787d620c84c0babb808ec33766e25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:51:06.352774 20619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d0214cc69f1481d8a44bd1064084aed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:06.356717 20964 tablet_bootstrap.cc:492] T 6d0214cc69f1481d8a44bd1064084aed P aa036f3224ca4c209ca0ee35a6b59fc6: Bootstrap starting.
I20260621 07:51:06.358935 20966 tablet_bootstrap.cc:492] T 6d0214cc69f1481d8a44bd1064084aed P 8a23f72ce2f14945ac5cfdfab3bbbec7: Bootstrap starting.
I20260621 07:51:06.359066 20964 tablet_bootstrap.cc:654] T 6d0214cc69f1481d8a44bd1064084aed P aa036f3224ca4c209ca0ee35a6b59fc6: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:06.359423 20965 tablet_bootstrap.cc:492] T 6d0214cc69f1481d8a44bd1064084aed P 80029e6fdf4b40a5bea1445789d65b1b: Bootstrap starting.
I20260621 07:51:06.359861 20964 log.cc:824] T 6d0214cc69f1481d8a44bd1064084aed P aa036f3224ca4c209ca0ee35a6b59fc6: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:06.360770 20966 tablet_bootstrap.cc:654] T 6d0214cc69f1481d8a44bd1064084aed P 8a23f72ce2f14945ac5cfdfab3bbbec7: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:06.361281 20965 tablet_bootstrap.cc:654] T 6d0214cc69f1481d8a44bd1064084aed P 80029e6fdf4b40a5bea1445789d65b1b: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:06.361390 20966 log.cc:824] T 6d0214cc69f1481d8a44bd1064084aed P 8a23f72ce2f14945ac5cfdfab3bbbec7: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:06.361799 20964 tablet_bootstrap.cc:492] T 6d0214cc69f1481d8a44bd1064084aed P aa036f3224ca4c209ca0ee35a6b59fc6: No bootstrap required, opened a new log
I20260621 07:51:06.362000 20964 ts_tablet_manager.cc:1397] T 6d0214cc69f1481d8a44bd1064084aed P aa036f3224ca4c209ca0ee35a6b59fc6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260621 07:51:06.361995 20965 log.cc:824] T 6d0214cc69f1481d8a44bd1064084aed P 80029e6fdf4b40a5bea1445789d65b1b: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:06.363152 20966 tablet_bootstrap.cc:492] T 6d0214cc69f1481d8a44bd1064084aed P 8a23f72ce2f14945ac5cfdfab3bbbec7: No bootstrap required, opened a new log
I20260621 07:51:06.363338 20966 ts_tablet_manager.cc:1397] T 6d0214cc69f1481d8a44bd1064084aed P 8a23f72ce2f14945ac5cfdfab3bbbec7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260621 07:51:06.363704 20965 tablet_bootstrap.cc:492] T 6d0214cc69f1481d8a44bd1064084aed P 80029e6fdf4b40a5bea1445789d65b1b: No bootstrap required, opened a new log
I20260621 07:51:06.363863 20965 ts_tablet_manager.cc:1397] T 6d0214cc69f1481d8a44bd1064084aed P 80029e6fdf4b40a5bea1445789d65b1b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260621 07:51:06.366009 20964 raft_consensus.cc:348] T 6d0214cc69f1481d8a44bd1064084aed P aa036f3224ca4c209ca0ee35a6b59fc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa036f3224ca4c209ca0ee35a6b59fc6" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 32927 } } peers { permanent_uuid: "8a23f72ce2f14945ac5cfdfab3bbbec7" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 35577 } } peers { permanent_uuid: "80029e6fdf4b40a5bea1445789d65b1b" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 46279 } }
I20260621 07:51:06.366257 20964 raft_consensus.cc:374] T 6d0214cc69f1481d8a44bd1064084aed P aa036f3224ca4c209ca0ee35a6b59fc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:06.366317 20964 raft_consensus.cc:729] T 6d0214cc69f1481d8a44bd1064084aed P aa036f3224ca4c209ca0ee35a6b59fc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa036f3224ca4c209ca0ee35a6b59fc6, State: Initialized, Role: FOLLOWER
I20260621 07:51:06.366451 20966 raft_consensus.cc:348] T 6d0214cc69f1481d8a44bd1064084aed P 8a23f72ce2f14945ac5cfdfab3bbbec7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa036f3224ca4c209ca0ee35a6b59fc6" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 32927 } } peers { permanent_uuid: "8a23f72ce2f14945ac5cfdfab3bbbec7" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 35577 } } peers { permanent_uuid: "80029e6fdf4b40a5bea1445789d65b1b" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 46279 } }
I20260621 07:51:06.366707 20966 raft_consensus.cc:374] T 6d0214cc69f1481d8a44bd1064084aed P 8a23f72ce2f14945ac5cfdfab3bbbec7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:06.366768 20966 raft_consensus.cc:729] T 6d0214cc69f1481d8a44bd1064084aed P 8a23f72ce2f14945ac5cfdfab3bbbec7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a23f72ce2f14945ac5cfdfab3bbbec7, State: Initialized, Role: FOLLOWER
I20260621 07:51:06.366755 20964 consensus_queue.cc:260] T 6d0214cc69f1481d8a44bd1064084aed P aa036f3224ca4c209ca0ee35a6b59fc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa036f3224ca4c209ca0ee35a6b59fc6" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 32927 } } peers { permanent_uuid: "8a23f72ce2f14945ac5cfdfab3bbbec7" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 35577 } } peers { permanent_uuid: "80029e6fdf4b40a5bea1445789d65b1b" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 46279 } }
I20260621 07:51:06.367161 20966 consensus_queue.cc:260] T 6d0214cc69f1481d8a44bd1064084aed P 8a23f72ce2f14945ac5cfdfab3bbbec7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa036f3224ca4c209ca0ee35a6b59fc6" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 32927 } } peers { permanent_uuid: "8a23f72ce2f14945ac5cfdfab3bbbec7" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 35577 } } peers { permanent_uuid: "80029e6fdf4b40a5bea1445789d65b1b" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 46279 } }
I20260621 07:51:06.367328 20816 heartbeater.cc:503] Master 127.0.192.254:39841 was elected leader, sending a full tablet report...
I20260621 07:51:06.367705 20965 raft_consensus.cc:348] T 6d0214cc69f1481d8a44bd1064084aed P 80029e6fdf4b40a5bea1445789d65b1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa036f3224ca4c209ca0ee35a6b59fc6" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 32927 } } peers { permanent_uuid: "8a23f72ce2f14945ac5cfdfab3bbbec7" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 35577 } } peers { permanent_uuid: "80029e6fdf4b40a5bea1445789d65b1b" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 46279 } }
I20260621 07:51:06.367848 20966 ts_tablet_manager.cc:1428] T 6d0214cc69f1481d8a44bd1064084aed P 8a23f72ce2f14945ac5cfdfab3bbbec7: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260621 07:51:06.367908 20965 raft_consensus.cc:374] T 6d0214cc69f1481d8a44bd1064084aed P 80029e6fdf4b40a5bea1445789d65b1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:06.367954 20965 raft_consensus.cc:729] T 6d0214cc69f1481d8a44bd1064084aed P 80029e6fdf4b40a5bea1445789d65b1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80029e6fdf4b40a5bea1445789d65b1b, State: Initialized, Role: FOLLOWER
I20260621 07:51:06.368136 20685 heartbeater.cc:503] Master 127.0.192.254:39841 was elected leader, sending a full tablet report...
I20260621 07:51:06.368275 20965 consensus_queue.cc:260] T 6d0214cc69f1481d8a44bd1064084aed P 80029e6fdf4b40a5bea1445789d65b1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa036f3224ca4c209ca0ee35a6b59fc6" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 32927 } } peers { permanent_uuid: "8a23f72ce2f14945ac5cfdfab3bbbec7" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 35577 } } peers { permanent_uuid: "80029e6fdf4b40a5bea1445789d65b1b" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 46279 } }
I20260621 07:51:06.368774 20964 ts_tablet_manager.cc:1428] T 6d0214cc69f1481d8a44bd1064084aed P aa036f3224ca4c209ca0ee35a6b59fc6: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260621 07:51:06.369027 20965 ts_tablet_manager.cc:1428] T 6d0214cc69f1481d8a44bd1064084aed P 80029e6fdf4b40a5bea1445789d65b1b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260621 07:51:06.369306 20947 heartbeater.cc:503] Master 127.0.192.254:39841 was elected leader, sending a full tablet report...
W20260621 07:51:06.395924 20817 tablet.cc:2369] T 6d0214cc69f1481d8a44bd1064084aed P aa036f3224ca4c209ca0ee35a6b59fc6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:51:06.418409 20971 raft_consensus.cc:482] T 6d0214cc69f1481d8a44bd1064084aed P 8a23f72ce2f14945ac5cfdfab3bbbec7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:51:06.418576 20971 raft_consensus.cc:504] T 6d0214cc69f1481d8a44bd1064084aed P 8a23f72ce2f14945ac5cfdfab3bbbec7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa036f3224ca4c209ca0ee35a6b59fc6" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 32927 } } peers { permanent_uuid: "8a23f72ce2f14945ac5cfdfab3bbbec7" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 35577 } } peers { permanent_uuid: "80029e6fdf4b40a5bea1445789d65b1b" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 46279 } }
I20260621 07:51:06.419637 20971 leader_election.cc:288] T 6d0214cc69f1481d8a44bd1064084aed P 8a23f72ce2f14945ac5cfdfab3bbbec7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa036f3224ca4c209ca0ee35a6b59fc6 (127.0.192.194:32927), 80029e6fdf4b40a5bea1445789d65b1b (127.0.192.195:46279)
I20260621 07:51:06.423575 20687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:06.419795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:32927 (local address 127.0.192.193:40477)
0621 07:51:06.419911 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:06.419927 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:06.420006 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:06.420229 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:06.420232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:06.420250 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:51:06.420372 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:06.420380 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:06.421352 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:06.421354 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:06.422458 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:06.422467 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:06.423157 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:06.423159 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:06.423226 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:06.423244 (+ 18us) client_negotiation.cc:769] Sending connection context
0621 07:51:06.423335 (+ 91us) client_negotiation.cc:240] Negotiation successful
0621 07:51:06.423419 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260621 07:51:06.423590 20963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:06.419904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:40477 (local address 127.0.192.194:32927)
0621 07:51:06.420024 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:06.420027 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:06.420043 (+ 16us) server_negotiation.cc:407] Connection header received
0621 07:51:06.420085 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:06.420086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:06.420123 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:06.420213 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:51:06.420488 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:06.421234 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:06.422581 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:06.423042 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:06.423158 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:06.423199 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:06.423382 (+ 183us) server_negotiation.cc:299] Negotiation successful
0621 07:51:06.423463 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260621 07:51:06.423914 20962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:06.420255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.192.193:45757 (local address 127.0.192.195:46279)
0621 07:51:06.420379 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:06.420382 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:06.420579 (+ 197us) server_negotiation.cc:407] Connection header received
0621 07:51:06.420714 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:06.420720 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:06.420766 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:06.420843 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:51:06.421069 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:06.421640 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:06.422846 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:06.423380 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:06.423477 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:06.423514 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:06.423725 (+ 211us) server_negotiation.cc:299] Negotiation successful
0621 07:51:06.423804 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 07:51:06.424156 20770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d0214cc69f1481d8a44bd1064084aed" candidate_uuid: "8a23f72ce2f14945ac5cfdfab3bbbec7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa036f3224ca4c209ca0ee35a6b59fc6" is_pre_election: true
I20260621 07:51:06.424235 20973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:06.420040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.195:46279 (local address 127.0.192.193:45757)
0621 07:51:06.420473 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:06.420487 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:06.420617 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:06.420860 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:06.420862 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:06.420874 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:51:06.420963 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:06.420966 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:06.421741 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:06.421745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:06.422740 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:06.422747 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:06.423497 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:06.423500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:06.423580 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:06.423607 (+ 27us) client_negotiation.cc:769] Sending connection context
0621 07:51:06.424037 (+ 430us) client_negotiation.cc:240] Negotiation successful
0621 07:51:06.424099 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":149,"threads_started":1}
I20260621 07:51:06.424427 20770 raft_consensus.cc:2454] T 6d0214cc69f1481d8a44bd1064084aed P aa036f3224ca4c209ca0ee35a6b59fc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a23f72ce2f14945ac5cfdfab3bbbec7 in term 0.
I20260621 07:51:06.424916 20575 leader_election.cc:302] T 6d0214cc69f1481d8a44bd1064084aed P 8a23f72ce2f14945ac5cfdfab3bbbec7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a23f72ce2f14945ac5cfdfab3bbbec7, aa036f3224ca4c209ca0ee35a6b59fc6; no voters:
I20260621 07:51:06.424954 20901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d0214cc69f1481d8a44bd1064084aed" candidate_uuid: "8a23f72ce2f14945ac5cfdfab3bbbec7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80029e6fdf4b40a5bea1445789d65b1b" is_pre_election: true
I20260621 07:51:06.425177 20901 raft_consensus.cc:2454] T 6d0214cc69f1481d8a44bd1064084aed P 80029e6fdf4b40a5bea1445789d65b1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a23f72ce2f14945ac5cfdfab3bbbec7 in term 0.
I20260621 07:51:06.425621 20971 raft_consensus.cc:2781] T 6d0214cc69f1481d8a44bd1064084aed P 8a23f72ce2f14945ac5cfdfab3bbbec7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:51:06.425729 20971 raft_consensus.cc:482] T 6d0214cc69f1481d8a44bd1064084aed P 8a23f72ce2f14945ac5cfdfab3bbbec7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:51:06.425804 20971 raft_consensus.cc:3037] T 6d0214cc69f1481d8a44bd1064084aed P 8a23f72ce2f14945ac5cfdfab3bbbec7 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:06.426846 20971 raft_consensus.cc:504] T 6d0214cc69f1481d8a44bd1064084aed P 8a23f72ce2f14945ac5cfdfab3bbbec7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa036f3224ca4c209ca0ee35a6b59fc6" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 32927 } } peers { permanent_uuid: "8a23f72ce2f14945ac5cfdfab3bbbec7" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 35577 } } peers { permanent_uuid: "80029e6fdf4b40a5bea1445789d65b1b" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 46279 } }
I20260621 07:51:06.427179 20971 leader_election.cc:288] T 6d0214cc69f1481d8a44bd1064084aed P 8a23f72ce2f14945ac5cfdfab3bbbec7 [CANDIDATE]: Term 1 election: Requested vote from peers aa036f3224ca4c209ca0ee35a6b59fc6 (127.0.192.194:32927), 80029e6fdf4b40a5bea1445789d65b1b (127.0.192.195:46279)
I20260621 07:51:06.427475 20770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d0214cc69f1481d8a44bd1064084aed" candidate_uuid: "8a23f72ce2f14945ac5cfdfab3bbbec7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa036f3224ca4c209ca0ee35a6b59fc6"
I20260621 07:51:06.427598 20770 raft_consensus.cc:3037] T 6d0214cc69f1481d8a44bd1064084aed P aa036f3224ca4c209ca0ee35a6b59fc6 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:06.427599 20901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d0214cc69f1481d8a44bd1064084aed" candidate_uuid: "8a23f72ce2f14945ac5cfdfab3bbbec7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80029e6fdf4b40a5bea1445789d65b1b"
I20260621 07:51:06.427735 20901 raft_consensus.cc:3037] T 6d0214cc69f1481d8a44bd1064084aed P 80029e6fdf4b40a5bea1445789d65b1b [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:06.428630 20770 raft_consensus.cc:2454] T 6d0214cc69f1481d8a44bd1064084aed P aa036f3224ca4c209ca0ee35a6b59fc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a23f72ce2f14945ac5cfdfab3bbbec7 in term 1.
I20260621 07:51:06.428821 20901 raft_consensus.cc:2454] T 6d0214cc69f1481d8a44bd1064084aed P 80029e6fdf4b40a5bea1445789d65b1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a23f72ce2f14945ac5cfdfab3bbbec7 in term 1.
I20260621 07:51:06.428960 20575 leader_election.cc:302] T 6d0214cc69f1481d8a44bd1064084aed P 8a23f72ce2f14945ac5cfdfab3bbbec7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a23f72ce2f14945ac5cfdfab3bbbec7, aa036f3224ca4c209ca0ee35a6b59fc6; no voters:
I20260621 07:51:06.429129 20971 raft_consensus.cc:2781] T 6d0214cc69f1481d8a44bd1064084aed P 8a23f72ce2f14945ac5cfdfab3bbbec7 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:06.429332 20971 raft_consensus.cc:686] T 6d0214cc69f1481d8a44bd1064084aed P 8a23f72ce2f14945ac5cfdfab3bbbec7 [term 1 LEADER]: Becoming Leader. State: Replica: 8a23f72ce2f14945ac5cfdfab3bbbec7, State: Running, Role: LEADER
I20260621 07:51:06.429631 20971 consensus_queue.cc:237] T 6d0214cc69f1481d8a44bd1064084aed P 8a23f72ce2f14945ac5cfdfab3bbbec7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa036f3224ca4c209ca0ee35a6b59fc6" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 32927 } } peers { permanent_uuid: "8a23f72ce2f14945ac5cfdfab3bbbec7" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 35577 } } peers { permanent_uuid: "80029e6fdf4b40a5bea1445789d65b1b" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 46279 } }
I20260621 07:51:06.432615 20494 catalog_manager.cc:5495] T 6d0214cc69f1481d8a44bd1064084aed P 8a23f72ce2f14945ac5cfdfab3bbbec7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a23f72ce2f14945ac5cfdfab3bbbec7 (127.0.192.193). New cstate: current_term: 1 leader_uuid: "8a23f72ce2f14945ac5cfdfab3bbbec7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa036f3224ca4c209ca0ee35a6b59fc6" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a23f72ce2f14945ac5cfdfab3bbbec7" member_type: VOTER last_known_addr { host: "127.0.192.193" port: 35577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80029e6fdf4b40a5bea1445789d65b1b" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 46279 } health_report { overall_health: UNKNOWN } } }
I20260621 07:51:06.443440 20960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:06.440532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33010 (local address 127.0.192.193:35577)
0621 07:51:06.440658 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:06.440661 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:06.440674 (+ 13us) server_negotiation.cc:407] Connection header received
0621 07:51:06.440769 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:06.440771 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:06.440816 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:06.440892 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:51:06.441074 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:06.441510 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:06.442283 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:06.442482 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:06.442546 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:06.442767 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:51:06.443101 (+ 334us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:51:06.443179 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:06.443232 (+ 53us) server_negotiation.cc:299] Negotiation successful
0621 07:51:06.443268 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 07:51:06.448838 20901 raft_consensus.cc:1261] T 6d0214cc69f1481d8a44bd1064084aed P 80029e6fdf4b40a5bea1445789d65b1b [term 1 FOLLOWER]: Refusing update from remote peer 8a23f72ce2f14945ac5cfdfab3bbbec7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:51:06.448854 20770 raft_consensus.cc:1261] T 6d0214cc69f1481d8a44bd1064084aed P aa036f3224ca4c209ca0ee35a6b59fc6 [term 1 FOLLOWER]: Refusing update from remote peer 8a23f72ce2f14945ac5cfdfab3bbbec7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:51:06.449466 20971 consensus_queue.cc:1035] T 6d0214cc69f1481d8a44bd1064084aed P 8a23f72ce2f14945ac5cfdfab3bbbec7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80029e6fdf4b40a5bea1445789d65b1b" member_type: VOTER last_known_addr { host: "127.0.192.195" port: 46279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:51:06.449635 20974 consensus_queue.cc:1035] T 6d0214cc69f1481d8a44bd1064084aed P 8a23f72ce2f14945ac5cfdfab3bbbec7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa036f3224ca4c209ca0ee35a6b59fc6" member_type: VOTER last_known_addr { host: "127.0.192.194" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:51:06.456965 20978 mvcc.cc:205] Tried to move back new op lower bound from 7299187779368587264 to 7299187779299061760. Current Snapshot: MvccSnapshot[applied={T|T < 7299187779368587264}]
I20260621 07:51:06.457221 20979 mvcc.cc:205] Tried to move back new op lower bound from 7299187779368587264 to 7299187779299061760. Current Snapshot: MvccSnapshot[applied={T|T < 7299187779368587264}]
I20260621 07:51:06.464395 20495 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:51:06.464543 20495 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:51:06.466745 20495 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 54f1b422929d418095ad5010754121e8: Sending DeleteTablet for 3 replicas of tablet 6d0214cc69f1481d8a44bd1064084aed
I20260621 07:51:06.467442 20619 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d0214cc69f1481d8a44bd1064084aed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:51:06 UTC) from {username='slave'} at 127.0.0.1:32996
I20260621 07:51:06.467510 20881 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d0214cc69f1481d8a44bd1064084aed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:51:06 UTC) from {username='slave'} at 127.0.0.1:45882
I20260621 07:51:06.467890 20988 tablet_replica.cc:330] T 6d0214cc69f1481d8a44bd1064084aed P 80029e6fdf4b40a5bea1445789d65b1b: stopping tablet replica
I20260621 07:51:06.468019 20750 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d0214cc69f1481d8a44bd1064084aed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:51:06 UTC) from {username='slave'} at 127.0.0.1:50630
I20260621 07:51:06.468261 20988 raft_consensus.cc:2229] T 6d0214cc69f1481d8a44bd1064084aed P 80029e6fdf4b40a5bea1445789d65b1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:51:06.468437 20989 tablet_replica.cc:330] T 6d0214cc69f1481d8a44bd1064084aed P aa036f3224ca4c209ca0ee35a6b59fc6: stopping tablet replica
I20260621 07:51:06.468523 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 20557
I20260621 07:51:06.469202 20988 raft_consensus.cc:2258] T 6d0214cc69f1481d8a44bd1064084aed P 80029e6fdf4b40a5bea1445789d65b1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:51:06.470671 20988 ts_tablet_manager.cc:1905] T 6d0214cc69f1481d8a44bd1064084aed P 80029e6fdf4b40a5bea1445789d65b1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:51:06.470656 20989 raft_consensus.cc:2229] T 6d0214cc69f1481d8a44bd1064084aed P aa036f3224ca4c209ca0ee35a6b59fc6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:51:06.470919 20989 raft_consensus.cc:2258] T 6d0214cc69f1481d8a44bd1064084aed P aa036f3224ca4c209ca0ee35a6b59fc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:51:06.472661 20989 ts_tablet_manager.cc:1905] T 6d0214cc69f1481d8a44bd1064084aed P aa036f3224ca4c209ca0ee35a6b59fc6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:51:06.473102 20988 ts_tablet_manager.cc:1918] T 6d0214cc69f1481d8a44bd1064084aed P 80029e6fdf4b40a5bea1445789d65b1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:51:06.473186 20988 log.cc:1196] T 6d0214cc69f1481d8a44bd1064084aed P 80029e6fdf4b40a5bea1445789d65b1b: Deleting WAL directory at /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782028028522580-771-0/minicluster-data/ts-2/wal/wals/6d0214cc69f1481d8a44bd1064084aed
I20260621 07:51:06.473482 20988 ts_tablet_manager.cc:1939] T 6d0214cc69f1481d8a44bd1064084aed P 80029e6fdf4b40a5bea1445789d65b1b: Deleting consensus metadata
I20260621 07:51:06.474560 20484 catalog_manager.cc:4841] TS 80029e6fdf4b40a5bea1445789d65b1b (127.0.192.195:46279): tablet 6d0214cc69f1481d8a44bd1064084aed (table test-table [id=992787d620c84c0babb808ec33766e25]) successfully deleted
W20260621 07:51:06.474947 20482 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.192.193:35577 (error 108)
I20260621 07:51:06.475128 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 20688
I20260621 07:51:06.476374 20961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:06.476129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.193:35577 (local address 127.0.0.1:33018)
0621 07:51:06.476245 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:06.476313 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.193:35577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260621 07:51:06.476502 20482 catalog_manager.cc:4584] TS 8a23f72ce2f14945ac5cfdfab3bbbec7 (127.0.192.193:35577): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6d0214cc69f1481d8a44bd1064084aed: Network error: Client connection negotiation failed: client connection to 127.0.192.193:35577: connect: Connection refused (error 111)
I20260621 07:51:06.479753 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 20819
I20260621 07:51:06.480150 20961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:06.479901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.194:32927 (local address 127.0.0.1:50636)
0621 07:51:06.480043 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:06.480087 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.194:32927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260621 07:51:06.484469 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 20466
2026-06-21T07:51:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (897 ms)
[----------] 6 tests from EncryptionPolicyTest (9555 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028028522580-771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:51:06 dist-test-slave-lzx3 krb5kdc[20993](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:51:06 dist-test-slave-lzx3 krb5kdc[20993](info): set up 2 sockets
Jun 21 07:51:06 dist-test-slave-lzx3 krb5kdc[20993](info): commencing operation
krb5kdc: starting...
W20260621 07:51:07.440346 771 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028028522580-771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028028522580-771-0/krb5kdc/test-user.keytab.
Jun 21 07:51:07 dist-test-slave-lzx3 krb5kdc[20993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:51:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:51:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 07:51:07.541250 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33205
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:07.638315 21006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:07.638528 21006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:07.638563 21006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:07.642746 21006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:51:07.642803 21006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:07.642822 21006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:07.642835 21006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:51:07.642849 21006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:51:07.647655 21006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33205
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:07.648435 21006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:07.649329 21006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:07.656948 21012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:07.657088 21006 server_base.cc:1034] running on GCE node
W20260621 07:51:07.656942 21014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:07.656942 21011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:07.657752 21006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:07.658648 21006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:07.659837 21006 hybrid_clock.cc:648] HybridClock initialized: now 1782028267659822 us; error 29 us; skew 500 ppm
Jun 21 07:51:07 dist-test-slave-lzx3 krb5kdc[20993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:07.662247 21006 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 07:51:07.662931 21006 webserver.cc:460] Webserver started at http://127.0.0.1:32991/ using document root <none> and password file <none>
I20260621 07:51:07.663365 21006 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:07.663419 21006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:07.663573 21006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:07.665447 21006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "5e27e9d4207e46c7b42a30359fa27909"
format_stamp: "Formatted at 2026-06-21 07:51:07 on dist-test-slave-lzx3"
server_key: "dab0e6d5b5542bdada3c85c5fc68bb9e"
server_key_iv: "978f19eeeb3fcf873b774e73841b87b1"
server_key_version: "encryptionkey@0"
I20260621 07:51:07.665938 21006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "5e27e9d4207e46c7b42a30359fa27909"
format_stamp: "Formatted at 2026-06-21 07:51:07 on dist-test-slave-lzx3"
server_key: "dab0e6d5b5542bdada3c85c5fc68bb9e"
server_key_iv: "978f19eeeb3fcf873b774e73841b87b1"
server_key_version: "encryptionkey@0"
I20260621 07:51:07.669227 21006 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260621 07:51:07.671391 21021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:07.672818 21006 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260621 07:51:07.672961 21006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "5e27e9d4207e46c7b42a30359fa27909"
format_stamp: "Formatted at 2026-06-21 07:51:07 on dist-test-slave-lzx3"
server_key: "dab0e6d5b5542bdada3c85c5fc68bb9e"
server_key_iv: "978f19eeeb3fcf873b774e73841b87b1"
server_key_version: "encryptionkey@0"
I20260621 07:51:07.673079 21006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:07.703712 21006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:07.706466 21006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:07.706624 21006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:07.712505 21006 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33205
I20260621 07:51:07.712599 21073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33205 every 8 connection(s)
I20260621 07:51:07.713349 21006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:51:07.715937 21074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:07.716197 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 21006
I20260621 07:51:07.716338 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:51:07.716689 771 external_mini_cluster.cc:1428] Setting key f09accff9f7e01f0f016afefd64291b4
I20260621 07:51:07.721818 21074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e27e9d4207e46c7b42a30359fa27909: Bootstrap starting.
I20260621 07:51:07.723917 21074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e27e9d4207e46c7b42a30359fa27909: Neither blocks nor log segments found. Creating new log.
Jun 21 07:51:07 dist-test-slave-lzx3 krb5kdc[20993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260621 07:51:07.724706 21074 log.cc:824] T 00000000000000000000000000000000 P 5e27e9d4207e46c7b42a30359fa27909: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:07.726727 21074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e27e9d4207e46c7b42a30359fa27909: No bootstrap required, opened a new log
I20260621 07:51:07.729628 21077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:07.717876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60557 (local address 127.0.0.1:33205)
0621 07:51:07.718253 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:07.718265 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:07.718297 (+ 32us) server_negotiation.cc:407] Connection header received
0621 07:51:07.719027 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:07.719044 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:07.719346 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:07.719661 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:07.720414 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:07.721360 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:07.722218 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:07.722631 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:07.722785 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:07.725274 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:07.725288 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:07.725302 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:07.725328 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:07.726938 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:07.727432 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:07.727438 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:07.727443 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:07.727555 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:07.727913 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:07.727916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:07.727918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:07.728275 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:07.728499 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:07.728794 (+ 295us) server_negotiation.cc:299] Negotiation successful
0621 07:51:07.729010 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":115,"threads_started":1}
I20260621 07:51:07.730265 21074 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e27e9d4207e46c7b42a30359fa27909 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e27e9d4207e46c7b42a30359fa27909" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33205 } }
I20260621 07:51:07.730458 21074 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e27e9d4207e46c7b42a30359fa27909 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:07.730535 21074 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e27e9d4207e46c7b42a30359fa27909 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e27e9d4207e46c7b42a30359fa27909, State: Initialized, Role: FOLLOWER
I20260621 07:51:07.730974 21074 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e27e9d4207e46c7b42a30359fa27909 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e27e9d4207e46c7b42a30359fa27909" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33205 } }
I20260621 07:51:07.731143 21074 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e27e9d4207e46c7b42a30359fa27909 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:51:07.731205 21074 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e27e9d4207e46c7b42a30359fa27909 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:51:07.731304 21074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e27e9d4207e46c7b42a30359fa27909 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:07.732303 21074 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e27e9d4207e46c7b42a30359fa27909 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e27e9d4207e46c7b42a30359fa27909" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33205 } }
I20260621 07:51:07.732497 771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:51:07.732674 21074 leader_election.cc:302] T 00000000000000000000000000000000 P 5e27e9d4207e46c7b42a30359fa27909 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e27e9d4207e46c7b42a30359fa27909; no voters:
I20260621 07:51:07.732978 21074 leader_election.cc:288] T 00000000000000000000000000000000 P 5e27e9d4207e46c7b42a30359fa27909 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:51:07.733103 21079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e27e9d4207e46c7b42a30359fa27909 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:07.733302 21079 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e27e9d4207e46c7b42a30359fa27909 [term 1 LEADER]: Becoming Leader. State: Replica: 5e27e9d4207e46c7b42a30359fa27909, State: Running, Role: LEADER
I20260621 07:51:07.733589 21079 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e27e9d4207e46c7b42a30359fa27909 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e27e9d4207e46c7b42a30359fa27909" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33205 } }
I20260621 07:51:07.734102 21074 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e27e9d4207e46c7b42a30359fa27909 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:51:07.734696 21079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e27e9d4207e46c7b42a30359fa27909 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e27e9d4207e46c7b42a30359fa27909. Latest consensus state: current_term: 1 leader_uuid: "5e27e9d4207e46c7b42a30359fa27909" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e27e9d4207e46c7b42a30359fa27909" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33205 } } }
I20260621 07:51:07.734824 21079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e27e9d4207e46c7b42a30359fa27909 [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:07.735312 21080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e27e9d4207e46c7b42a30359fa27909 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e27e9d4207e46c7b42a30359fa27909" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e27e9d4207e46c7b42a30359fa27909" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33205 } } }
I20260621 07:51:07.735498 21080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e27e9d4207e46c7b42a30359fa27909 [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:07.736367 21092 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:51:07.739259 21092 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:51:07.742735 21077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:07.734663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46347 (local address 127.0.0.1:33205)
0621 07:51:07.734766 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:07.734771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:07.734982 (+ 211us) server_negotiation.cc:407] Connection header received
0621 07:51:07.735166 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:07.735168 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:07.735212 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:07.735279 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:07.736144 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:07.736995 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:07.737818 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:07.738049 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:07.738156 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:07.739497 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:07.739499 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:07.739501 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:07.739519 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:07.740989 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:07.741488 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:07.741491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:07.741493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:07.741571 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:07.741898 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:07.741901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:07.741902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:07.742089 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:07.742221 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:07.742509 (+ 288us) server_negotiation.cc:299] Negotiation successful
0621 07:51:07.742596 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260621 07:51:07.744879 21092 catalog_manager.cc:1347] Generated new cluster ID: 78cccedcd1e546b48fd0e6987666d5a3
I20260621 07:51:07.744941 21092 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:51:07.754408 21092 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:51:07.755319 21092 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:51:07.766832 21092 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e27e9d4207e46c7b42a30359fa27909: Generated new TSK 0
I20260621 07:51:07.767395 21092 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:51:07.778112 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 21006
W20260621 07:51:07.782788 20991 connection.cc:535] client connection to 127.0.0.1:33205 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33205): Connection reset by peer (error 104)
2026-06-21T07:51:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1290 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028028522580-771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:51:07 dist-test-slave-lzx3 krb5kdc[21105](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:51:07 dist-test-slave-lzx3 krb5kdc[21105](info): set up 2 sockets
Jun 21 07:51:07 dist-test-slave-lzx3 krb5kdc[21105](info): commencing operation
krb5kdc: starting...
W20260621 07:51:08.712123 771 mini_kdc.cc:121] Time spent starting KDC: real 0.916s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028028522580-771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028028522580-771-0/krb5kdc/test-user.keytab.
Jun 21 07:51:08 dist-test-slave-lzx3 krb5kdc[21105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:51:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:51:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 07:51:08.813263 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35951
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35951
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:08.906358 21118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:08.906569 21118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:08.906603 21118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:08.910063 21118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:51:08.910106 21118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:08.910125 21118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:08.910138 21118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:51:08.910152 21118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:51:08.914340 21118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35951
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:08.915092 21118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:08.915899 21118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:08.923317 21123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:08.923446 21118 server_base.cc:1034] running on GCE node
W20260621 07:51:08.923362 21124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:08.923327 21126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:08.924064 21118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:08.925141 21118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:08.926319 21118 hybrid_clock.cc:648] HybridClock initialized: now 1782028268926295 us; error 36 us; skew 500 ppm
Jun 21 07:51:08 dist-test-slave-lzx3 krb5kdc[21105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:08.928683 21118 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 07:51:08.929414 21118 webserver.cc:460] Webserver started at http://127.0.0.1:43001/ using document root <none> and password file <none>
I20260621 07:51:08.929862 21118 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:08.929958 21118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:08.930167 21118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:08.931866 21118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "4dbb87cf15d14ae2a76782bbcde2ca9a"
format_stamp: "Formatted at 2026-06-21 07:51:08 on dist-test-slave-lzx3"
server_key: "8840a053199df4ffb2636248cedc3957"
server_key_iv: "4adc90f066078fb65b6f8d43089f79e9"
server_key_version: "encryptionkey@0"
I20260621 07:51:08.932349 21118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "4dbb87cf15d14ae2a76782bbcde2ca9a"
format_stamp: "Formatted at 2026-06-21 07:51:08 on dist-test-slave-lzx3"
server_key: "8840a053199df4ffb2636248cedc3957"
server_key_iv: "4adc90f066078fb65b6f8d43089f79e9"
server_key_version: "encryptionkey@0"
I20260621 07:51:08.935628 21118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260621 07:51:08.937902 21133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:08.938944 21118 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 07:51:08.939071 21118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "4dbb87cf15d14ae2a76782bbcde2ca9a"
format_stamp: "Formatted at 2026-06-21 07:51:08 on dist-test-slave-lzx3"
server_key: "8840a053199df4ffb2636248cedc3957"
server_key_iv: "4adc90f066078fb65b6f8d43089f79e9"
server_key_version: "encryptionkey@0"
I20260621 07:51:08.939172 21118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:08.958009 21118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:08.960167 21118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:08.960331 21118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:08.966338 21118 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35951
I20260621 07:51:08.966361 21185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35951 every 8 connection(s)
I20260621 07:51:08.967080 21118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:51:08.967535 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 21118
I20260621 07:51:08.967665 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:51:08.967906 771 external_mini_cluster.cc:1428] Setting key a26a8a7933b7ded598494862e4f6137d
I20260621 07:51:08.970283 21186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:08.975175 21189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:08.969007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59383 (local address 127.0.0.1:35951)
0621 07:51:08.969327 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:08.969340 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:08.969370 (+ 30us) server_negotiation.cc:407] Connection header received
0621 07:51:08.969896 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:08.969909 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:08.970177 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:08.970485 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:08.970727 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:08.971518 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:08.972617 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:08.972987 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:08.973085 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:08.973808 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:08.973812 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:08.973822 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:51:08.974120 (+ 298us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:08.974161 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:51:08.974179 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:08.974261 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:08.974454 (+ 193us) server_negotiation.cc:299] Negotiation successful
0621 07:51:08.974582 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":91,"threads_started":1}
I20260621 07:51:08.975674 21186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dbb87cf15d14ae2a76782bbcde2ca9a: Bootstrap starting.
I20260621 07:51:08.977761 771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:51:08.977882 21186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4dbb87cf15d14ae2a76782bbcde2ca9a: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:08.978600 21186 log.cc:824] T 00000000000000000000000000000000 P 4dbb87cf15d14ae2a76782bbcde2ca9a: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:08.980482 21186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dbb87cf15d14ae2a76782bbcde2ca9a: No bootstrap required, opened a new log
I20260621 07:51:08.983124 21189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:08.979748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48537 (local address 127.0.0.1:35951)
0621 07:51:08.979838 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:08.979841 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:08.979956 (+ 115us) server_negotiation.cc:407] Connection header received
0621 07:51:08.980047 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:08.980048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:08.980075 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:08.980126 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:08.980353 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:08.980859 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:08.981789 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:08.981951 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:08.982011 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:08.982727 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:08.982730 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:08.982732 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:51:08.982741 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:08.982759 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:51:08.982767 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:08.982822 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:08.982966 (+ 144us) server_negotiation.cc:299] Negotiation successful
0621 07:51:08.982988 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260621 07:51:08.984416 21186 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4dbb87cf15d14ae2a76782bbcde2ca9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dbb87cf15d14ae2a76782bbcde2ca9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35951 } }
I20260621 07:51:08.984664 21186 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4dbb87cf15d14ae2a76782bbcde2ca9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:08.984712 21186 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4dbb87cf15d14ae2a76782bbcde2ca9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dbb87cf15d14ae2a76782bbcde2ca9a, State: Initialized, Role: FOLLOWER
I20260621 07:51:08.985044 21186 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4dbb87cf15d14ae2a76782bbcde2ca9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dbb87cf15d14ae2a76782bbcde2ca9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35951 } }
I20260621 07:51:08.985152 21186 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4dbb87cf15d14ae2a76782bbcde2ca9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:51:08.985195 21186 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4dbb87cf15d14ae2a76782bbcde2ca9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:51:08.985258 21186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4dbb87cf15d14ae2a76782bbcde2ca9a [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:08.985987 21186 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4dbb87cf15d14ae2a76782bbcde2ca9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dbb87cf15d14ae2a76782bbcde2ca9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35951 } }
I20260621 07:51:08.986259 21186 leader_election.cc:302] T 00000000000000000000000000000000 P 4dbb87cf15d14ae2a76782bbcde2ca9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4dbb87cf15d14ae2a76782bbcde2ca9a; no voters:
I20260621 07:51:08.986457 21186 leader_election.cc:288] T 00000000000000000000000000000000 P 4dbb87cf15d14ae2a76782bbcde2ca9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:51:08.986553 21196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4dbb87cf15d14ae2a76782bbcde2ca9a [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:08.986768 21196 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4dbb87cf15d14ae2a76782bbcde2ca9a [term 1 LEADER]: Becoming Leader. State: Replica: 4dbb87cf15d14ae2a76782bbcde2ca9a, State: Running, Role: LEADER
I20260621 07:51:08.987005 21196 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4dbb87cf15d14ae2a76782bbcde2ca9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dbb87cf15d14ae2a76782bbcde2ca9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35951 } }
I20260621 07:51:08.987408 21186 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4dbb87cf15d14ae2a76782bbcde2ca9a [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:51:08.988421 21198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dbb87cf15d14ae2a76782bbcde2ca9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4dbb87cf15d14ae2a76782bbcde2ca9a. Latest consensus state: current_term: 1 leader_uuid: "4dbb87cf15d14ae2a76782bbcde2ca9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dbb87cf15d14ae2a76782bbcde2ca9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35951 } } }
I20260621 07:51:08.988531 21198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dbb87cf15d14ae2a76782bbcde2ca9a [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:08.988834 21197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dbb87cf15d14ae2a76782bbcde2ca9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4dbb87cf15d14ae2a76782bbcde2ca9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dbb87cf15d14ae2a76782bbcde2ca9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35951 } } }
I20260621 07:51:08.988941 21197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dbb87cf15d14ae2a76782bbcde2ca9a [sys.catalog]: This master's current role is: LEADER
W20260621 07:51:08.991559 21211 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4dbb87cf15d14ae2a76782bbcde2ca9a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:51:08.991632 21211 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:51:08.991706 21205 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:51:08.992304 21205 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:51:08.997313 21205 catalog_manager.cc:1347] Generated new cluster ID: fb30f6ec3b194bf2aac1c21f3b58a9b3
I20260621 07:51:08.997373 21205 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:51:09.002009 21205 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:51:09.002699 21205 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:51:09.011247 21205 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4dbb87cf15d14ae2a76782bbcde2ca9a: Generated new TSK 0
I20260621 07:51:09.011814 21205 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:51:09.019970 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 21118
W20260621 07:51:09.025027 21103 connection.cc:535] client connection to 127.0.0.1:35951 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35951): Connection reset by peer (error 104)
2026-06-21T07:51:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1242 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028028522580-771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:51:09 dist-test-slave-lzx3 krb5kdc[21217](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:51:09 dist-test-slave-lzx3 krb5kdc[21217](info): set up 2 sockets
Jun 21 07:51:09 dist-test-slave-lzx3 krb5kdc[21217](info): commencing operation
krb5kdc: starting...
W20260621 07:51:09.964318 771 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028028522580-771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028028522580-771-0/krb5kdc/test-user.keytab.
Jun 21 07:51:10 dist-test-slave-lzx3 krb5kdc[21217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:51:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:51:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 07:51:10.063709 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45569
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:10.156759 21230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:10.156937 21230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:10.156971 21230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:10.160368 21230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:51:10.160408 21230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:10.160425 21230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:10.160439 21230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:51:10.160451 21230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:51:10.164878 21230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45569
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:10.165632 21230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:10.166334 21230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:10.173044 21235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:10.173147 21230 server_base.cc:1034] running on GCE node
W20260621 07:51:10.173044 21236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:10.173026 21238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:10.173791 21230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:10.174727 21230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:10.175936 21230 hybrid_clock.cc:648] HybridClock initialized: now 1782028270175922 us; error 48 us; skew 500 ppm
Jun 21 07:51:10 dist-test-slave-lzx3 krb5kdc[21217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:10.182797 21230 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 07:51:10.183594 21230 webserver.cc:460] Webserver started at http://127.0.0.1:46735/ using document root <none> and password file <none>
I20260621 07:51:10.184092 21230 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:10.184185 21230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:10.184423 21230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:10.186280 21230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "e0b77607f0784235b20f2ded821a8eb0"
format_stamp: "Formatted at 2026-06-21 07:51:10 on dist-test-slave-lzx3"
server_key: "a5a28796e533e8b7f921aa5c46d3112f"
server_key_iv: "aca6813da122aca3f2210dcfba9ff488"
server_key_version: "encryptionkey@0"
I20260621 07:51:10.186779 21230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "e0b77607f0784235b20f2ded821a8eb0"
format_stamp: "Formatted at 2026-06-21 07:51:10 on dist-test-slave-lzx3"
server_key: "a5a28796e533e8b7f921aa5c46d3112f"
server_key_iv: "aca6813da122aca3f2210dcfba9ff488"
server_key_version: "encryptionkey@0"
I20260621 07:51:10.189903 21230 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260621 07:51:10.191942 21245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:10.192978 21230 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:51:10.193106 21230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "e0b77607f0784235b20f2ded821a8eb0"
format_stamp: "Formatted at 2026-06-21 07:51:10 on dist-test-slave-lzx3"
server_key: "a5a28796e533e8b7f921aa5c46d3112f"
server_key_iv: "aca6813da122aca3f2210dcfba9ff488"
server_key_version: "encryptionkey@0"
I20260621 07:51:10.193205 21230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:10.197448 21230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:10.199854 21230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:10.200038 21230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:10.206408 21230 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45569
I20260621 07:51:10.206418 21297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45569 every 8 connection(s)
I20260621 07:51:10.207175 21230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:51:10.207940 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 21230
I20260621 07:51:10.208192 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:51:10.208493 771 external_mini_cluster.cc:1428] Setting key 8f88adbccf19c29dd30b80766cf93b05
I20260621 07:51:10.209985 21298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:10.213302 21301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:10.209815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41295 (local address 127.0.0.1:45569)
0621 07:51:10.210181 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:10.210212 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:10.210248 (+ 36us) server_negotiation.cc:407] Connection header received
0621 07:51:10.210919 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:10.210936 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:10.211210 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:10.211498 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:10.212132 (+ 634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:10.212138 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:10.212151 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:51:10.212161 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:10.212216 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:51:10.212238 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:10.212361 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:10.212421 (+ 60us) server_negotiation.cc:299] Negotiation successful
0621 07:51:10.212541 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":102,"threads_started":1}
I20260621 07:51:10.215763 771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:51:10.215801 21298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0b77607f0784235b20f2ded821a8eb0: Bootstrap starting.
I20260621 07:51:10.217760 21298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0b77607f0784235b20f2ded821a8eb0: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:10.218454 21298 log.cc:824] T 00000000000000000000000000000000 P e0b77607f0784235b20f2ded821a8eb0: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:10.219456 21301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:10.217753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48227 (local address 127.0.0.1:45569)
0621 07:51:10.217864 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:10.217871 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:10.218094 (+ 223us) server_negotiation.cc:407] Connection header received
0621 07:51:10.218245 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:10.218249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:10.218288 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:10.218377 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:10.219060 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:10.219064 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:10.219066 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:51:10.219079 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:10.219106 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:51:10.219119 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:10.219224 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:10.219315 (+ 91us) server_negotiation.cc:299] Negotiation successful
0621 07:51:10.219358 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260621 07:51:10.220368 21298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0b77607f0784235b20f2ded821a8eb0: No bootstrap required, opened a new log
I20260621 07:51:10.223491 21298 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0b77607f0784235b20f2ded821a8eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b77607f0784235b20f2ded821a8eb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45569 } }
I20260621 07:51:10.223692 21298 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0b77607f0784235b20f2ded821a8eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:10.223781 21298 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0b77607f0784235b20f2ded821a8eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b77607f0784235b20f2ded821a8eb0, State: Initialized, Role: FOLLOWER
I20260621 07:51:10.224186 21298 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0b77607f0784235b20f2ded821a8eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b77607f0784235b20f2ded821a8eb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45569 } }
I20260621 07:51:10.224332 21298 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0b77607f0784235b20f2ded821a8eb0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:51:10.224403 21298 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0b77607f0784235b20f2ded821a8eb0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:51:10.224498 21298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0b77607f0784235b20f2ded821a8eb0 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:10.225337 21298 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0b77607f0784235b20f2ded821a8eb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b77607f0784235b20f2ded821a8eb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45569 } }
I20260621 07:51:10.225659 21298 leader_election.cc:302] T 00000000000000000000000000000000 P e0b77607f0784235b20f2ded821a8eb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0b77607f0784235b20f2ded821a8eb0; no voters:
I20260621 07:51:10.225939 21298 leader_election.cc:288] T 00000000000000000000000000000000 P e0b77607f0784235b20f2ded821a8eb0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:51:10.226052 21308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0b77607f0784235b20f2ded821a8eb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:10.226264 21308 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0b77607f0784235b20f2ded821a8eb0 [term 1 LEADER]: Becoming Leader. State: Replica: e0b77607f0784235b20f2ded821a8eb0, State: Running, Role: LEADER
I20260621 07:51:10.226539 21308 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0b77607f0784235b20f2ded821a8eb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b77607f0784235b20f2ded821a8eb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45569 } }
I20260621 07:51:10.227001 21298 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0b77607f0784235b20f2ded821a8eb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:51:10.228070 21310 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0b77607f0784235b20f2ded821a8eb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0b77607f0784235b20f2ded821a8eb0. Latest consensus state: current_term: 1 leader_uuid: "e0b77607f0784235b20f2ded821a8eb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b77607f0784235b20f2ded821a8eb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45569 } } }
I20260621 07:51:10.228204 21310 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0b77607f0784235b20f2ded821a8eb0 [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:10.228420 21309 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0b77607f0784235b20f2ded821a8eb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0b77607f0784235b20f2ded821a8eb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b77607f0784235b20f2ded821a8eb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45569 } } }
I20260621 07:51:10.228507 21309 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0b77607f0784235b20f2ded821a8eb0 [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:10.228884 21317 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:51:10.231050 21317 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:51:10.235338 21317 catalog_manager.cc:1347] Generated new cluster ID: 1c438e82482e4d12873b452da3921bf6
I20260621 07:51:10.235404 21317 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:51:10.255481 21317 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:51:10.255743 21317 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:51:10.263502 21317 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0b77607f0784235b20f2ded821a8eb0: Generated new TSK 0
I20260621 07:51:10.264039 21317 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:51:10.266551 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 21230
2026-06-21T07:51:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1245 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028028522580-771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:51:10 dist-test-slave-lzx3 krb5kdc[21329](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:51:10 dist-test-slave-lzx3 krb5kdc[21329](info): set up 2 sockets
Jun 21 07:51:10 dist-test-slave-lzx3 krb5kdc[21329](info): commencing operation
krb5kdc: starting...
W20260621 07:51:11.220268 771 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028028522580-771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028028522580-771-0/krb5kdc/test-user.keytab.
Jun 21 07:51:11 dist-test-slave-lzx3 krb5kdc[21329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:51:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:51:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 07:51:11.322975 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40999
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:11.417452 21342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:11.417675 21342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:11.417754 21342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:11.421646 21342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:51:11.421698 21342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:11.421753 21342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:11.421811 21342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:51:11.421855 21342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:51:11.427047 21342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:11.427932 21342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:11.428876 21342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:11.436378 21347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:11.436384 21350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:11.436403 21348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:11.436647 21342 server_base.cc:1034] running on GCE node
I20260621 07:51:11.437140 21342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:11.438108 21342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:11.439296 21342 hybrid_clock.cc:648] HybridClock initialized: now 1782028271439276 us; error 38 us; skew 500 ppm
Jun 21 07:51:11 dist-test-slave-lzx3 krb5kdc[21329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:11.441704 21342 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 07:51:11.442428 21342 webserver.cc:460] Webserver started at http://127.0.0.1:41055/ using document root <none> and password file <none>
I20260621 07:51:11.442871 21342 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:11.442924 21342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:11.443064 21342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:11.444895 21342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "bcc7105b18fc4186a5c45c36e97da459"
format_stamp: "Formatted at 2026-06-21 07:51:11 on dist-test-slave-lzx3"
server_key: "7907c54d0f1ec3a18746f6dd570c192f"
server_key_iv: "1d2d5040f1b6dee41fd0f4942c81e871"
server_key_version: "encryptionkey@0"
I20260621 07:51:11.445372 21342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "bcc7105b18fc4186a5c45c36e97da459"
format_stamp: "Formatted at 2026-06-21 07:51:11 on dist-test-slave-lzx3"
server_key: "7907c54d0f1ec3a18746f6dd570c192f"
server_key_iv: "1d2d5040f1b6dee41fd0f4942c81e871"
server_key_version: "encryptionkey@0"
I20260621 07:51:11.448544 21342 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:51:11.450744 21357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:11.451805 21342 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260621 07:51:11.451921 21342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "bcc7105b18fc4186a5c45c36e97da459"
format_stamp: "Formatted at 2026-06-21 07:51:11 on dist-test-slave-lzx3"
server_key: "7907c54d0f1ec3a18746f6dd570c192f"
server_key_iv: "1d2d5040f1b6dee41fd0f4942c81e871"
server_key_version: "encryptionkey@0"
I20260621 07:51:11.452016 21342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:11.472962 21342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:11.475301 21342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:11.475468 21342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:11.481891 21342 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40999
I20260621 07:51:11.481920 21409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40999 every 8 connection(s)
I20260621 07:51:11.482774 21342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:51:11.485387 21410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:11.487649 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 21342
I20260621 07:51:11.487782 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:51:11.488041 771 external_mini_cluster.cc:1428] Setting key 532def672534e98bad6cdcf77d263305
I20260621 07:51:11.490944 21410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcc7105b18fc4186a5c45c36e97da459: Bootstrap starting.
I20260621 07:51:11.493294 21410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bcc7105b18fc4186a5c45c36e97da459: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:11.494056 21410 log.cc:824] T 00000000000000000000000000000000 P bcc7105b18fc4186a5c45c36e97da459: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:11.495697 21410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcc7105b18fc4186a5c45c36e97da459: No bootstrap required, opened a new log
Jun 21 07:51:11 dist-test-slave-lzx3 krb5kdc[21329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260621 07:51:11.499194 21410 raft_consensus.cc:348] T 00000000000000000000000000000000 P bcc7105b18fc4186a5c45c36e97da459 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcc7105b18fc4186a5c45c36e97da459" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40999 } }
I20260621 07:51:11.499429 21410 raft_consensus.cc:374] T 00000000000000000000000000000000 P bcc7105b18fc4186a5c45c36e97da459 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:11.499506 21410 raft_consensus.cc:729] T 00000000000000000000000000000000 P bcc7105b18fc4186a5c45c36e97da459 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcc7105b18fc4186a5c45c36e97da459, State: Initialized, Role: FOLLOWER
I20260621 07:51:11.499963 21410 consensus_queue.cc:260] T 00000000000000000000000000000000 P bcc7105b18fc4186a5c45c36e97da459 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcc7105b18fc4186a5c45c36e97da459" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40999 } }
I20260621 07:51:11.500141 21410 raft_consensus.cc:388] T 00000000000000000000000000000000 P bcc7105b18fc4186a5c45c36e97da459 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:51:11.500208 21410 raft_consensus.cc:482] T 00000000000000000000000000000000 P bcc7105b18fc4186a5c45c36e97da459 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:51:11.500321 21410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bcc7105b18fc4186a5c45c36e97da459 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:11.501282 21410 raft_consensus.cc:504] T 00000000000000000000000000000000 P bcc7105b18fc4186a5c45c36e97da459 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcc7105b18fc4186a5c45c36e97da459" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40999 } }
I20260621 07:51:11.501636 21410 leader_election.cc:302] T 00000000000000000000000000000000 P bcc7105b18fc4186a5c45c36e97da459 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bcc7105b18fc4186a5c45c36e97da459; no voters:
I20260621 07:51:11.501955 21410 leader_election.cc:288] T 00000000000000000000000000000000 P bcc7105b18fc4186a5c45c36e97da459 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:51:11.502076 21415 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bcc7105b18fc4186a5c45c36e97da459 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:11.502341 21415 raft_consensus.cc:686] T 00000000000000000000000000000000 P bcc7105b18fc4186a5c45c36e97da459 [term 1 LEADER]: Becoming Leader. State: Replica: bcc7105b18fc4186a5c45c36e97da459, State: Running, Role: LEADER
I20260621 07:51:11.502696 21415 consensus_queue.cc:237] T 00000000000000000000000000000000 P bcc7105b18fc4186a5c45c36e97da459 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcc7105b18fc4186a5c45c36e97da459" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40999 } }
I20260621 07:51:11.503075 21413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:11.489263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42237 (local address 127.0.0.1:40999)
0621 07:51:11.489669 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:11.489691 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:11.489728 (+ 37us) server_negotiation.cc:407] Connection header received
0621 07:51:11.490440 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:11.490460 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:11.490822 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:11.491143 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:11.491740 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:11.493056 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:11.493963 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:11.494587 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:11.494765 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:11.497483 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:11.497488 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:11.497498 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:11.497523 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:11.499797 (+ 2274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:11.500293 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:11.500301 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:11.500307 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:11.500482 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:11.500793 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:11.500795 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:11.500796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:11.501138 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:11.501290 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:11.501920 (+ 630us) server_negotiation.cc:299] Negotiation successful
0621 07:51:11.502203 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":131,"threads_started":1}
I20260621 07:51:11.503144 21410 sys_catalog.cc:564] T 00000000000000000000000000000000 P bcc7105b18fc4186a5c45c36e97da459 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:51:11.504220 21417 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcc7105b18fc4186a5c45c36e97da459 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcc7105b18fc4186a5c45c36e97da459. Latest consensus state: current_term: 1 leader_uuid: "bcc7105b18fc4186a5c45c36e97da459" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcc7105b18fc4186a5c45c36e97da459" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40999 } } }
I20260621 07:51:11.504336 21417 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcc7105b18fc4186a5c45c36e97da459 [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:11.504683 21416 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcc7105b18fc4186a5c45c36e97da459 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bcc7105b18fc4186a5c45c36e97da459" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcc7105b18fc4186a5c45c36e97da459" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40999 } } }
I20260621 07:51:11.504765 21416 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcc7105b18fc4186a5c45c36e97da459 [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:11.506250 771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260621 07:51:11.507835 21430 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bcc7105b18fc4186a5c45c36e97da459: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:51:11.507910 21430 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:51:11.507969 21424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:51:11.508951 21424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:51:11.514868 21424 catalog_manager.cc:1347] Generated new cluster ID: 19acf28a9f75410ab0ef90026c1fef90
I20260621 07:51:11.514938 21424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:51:11.518666 21413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:11.508411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60751 (local address 127.0.0.1:40999)
0621 07:51:11.508506 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:11.508510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:11.508825 (+ 315us) server_negotiation.cc:407] Connection header received
0621 07:51:11.508956 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:11.508959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:11.509002 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:11.509092 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:11.509875 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:11.510683 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:11.511825 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:11.512198 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:11.512301 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:11.513799 (+ 1498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:11.513803 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:11.513806 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:11.513833 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:11.517092 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:11.517515 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:11.517521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:11.517523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:11.517601 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:11.517884 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:11.517887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:11.517888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:11.518046 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:11.518182 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:11.518400 (+ 218us) server_negotiation.cc:299] Negotiation successful
0621 07:51:11.518505 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260621 07:51:11.521131 21424 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:51:11.522132 21424 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:51:11.528674 21424 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bcc7105b18fc4186a5c45c36e97da459: Generated new TSK 0
I20260621 07:51:11.529233 21424 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:51:11.535395 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 21342
2026-06-21T07:51:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1268 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028028522580-771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:51:11 dist-test-slave-lzx3 krb5kdc[21441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:51:11 dist-test-slave-lzx3 krb5kdc[21441](info): set up 2 sockets
Jun 21 07:51:11 dist-test-slave-lzx3 krb5kdc[21441](info): commencing operation
krb5kdc: starting...
W20260621 07:51:12.468039 771 mini_kdc.cc:121] Time spent starting KDC: real 0.914s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028028522580-771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028028522580-771-0/krb5kdc/test-user.keytab.
Jun 21 07:51:12 dist-test-slave-lzx3 krb5kdc[21441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:51:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:51:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 07:51:12.570825 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42103
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:12.667599 21454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:12.667826 21454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:12.667888 21454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:12.671217 21454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:51:12.671294 21454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:12.671332 21454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:12.671386 21454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:51:12.671432 21454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:51:12.675951 21454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:12.676743 21454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:12.677542 21454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:12.684787 21462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:12.684795 21459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:12.684830 21460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:12.684983 21454 server_base.cc:1034] running on GCE node
I20260621 07:51:12.685523 21454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:12.686391 21454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:12.687597 21454 hybrid_clock.cc:648] HybridClock initialized: now 1782028272687578 us; error 33 us; skew 500 ppm
Jun 21 07:51:12 dist-test-slave-lzx3 krb5kdc[21441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028272, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:12.689859 21454 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 07:51:12.690522 21454 webserver.cc:460] Webserver started at http://127.0.0.1:45913/ using document root <none> and password file <none>
I20260621 07:51:12.690958 21454 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:12.691035 21454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:12.691220 21454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:12.692828 21454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "78d3cf0a2dd6463eb00ba8ef9019dba2"
format_stamp: "Formatted at 2026-06-21 07:51:12 on dist-test-slave-lzx3"
server_key: "ca881690eb0c4e5ccdf3030517afe0c1"
server_key_iv: "ee8bf4a48e7a0e9c33edac22dd1ca097"
server_key_version: "encryptionkey@0"
I20260621 07:51:12.693286 21454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "78d3cf0a2dd6463eb00ba8ef9019dba2"
format_stamp: "Formatted at 2026-06-21 07:51:12 on dist-test-slave-lzx3"
server_key: "ca881690eb0c4e5ccdf3030517afe0c1"
server_key_iv: "ee8bf4a48e7a0e9c33edac22dd1ca097"
server_key_version: "encryptionkey@0"
I20260621 07:51:12.696307 21454 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:51:12.698436 21469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:12.699404 21454 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:51:12.699527 21454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "78d3cf0a2dd6463eb00ba8ef9019dba2"
format_stamp: "Formatted at 2026-06-21 07:51:12 on dist-test-slave-lzx3"
server_key: "ca881690eb0c4e5ccdf3030517afe0c1"
server_key_iv: "ee8bf4a48e7a0e9c33edac22dd1ca097"
server_key_version: "encryptionkey@0"
I20260621 07:51:12.699622 21454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:12.715662 21454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:12.718385 21454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:12.718571 21454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:12.724778 21454 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42103
I20260621 07:51:12.724833 21521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42103 every 8 connection(s)
I20260621 07:51:12.725637 21454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:51:12.728004 21522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:12.732766 21522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d3cf0a2dd6463eb00ba8ef9019dba2: Bootstrap starting.
I20260621 07:51:12.734983 21522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78d3cf0a2dd6463eb00ba8ef9019dba2: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:12.735500 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 21454
I20260621 07:51:12.735647 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:51:12.735849 21522 log.cc:824] T 00000000000000000000000000000000 P 78d3cf0a2dd6463eb00ba8ef9019dba2: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:12.735944 771 external_mini_cluster.cc:1428] Setting key e0a23cbac1266476e7d9292f3d85caeb
I20260621 07:51:12.737671 21522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d3cf0a2dd6463eb00ba8ef9019dba2: No bootstrap required, opened a new log
I20260621 07:51:12.741205 21522 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78d3cf0a2dd6463eb00ba8ef9019dba2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d3cf0a2dd6463eb00ba8ef9019dba2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42103 } }
I20260621 07:51:12.741441 21522 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78d3cf0a2dd6463eb00ba8ef9019dba2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:12.741547 21522 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78d3cf0a2dd6463eb00ba8ef9019dba2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78d3cf0a2dd6463eb00ba8ef9019dba2, State: Initialized, Role: FOLLOWER
I20260621 07:51:12.742046 21522 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78d3cf0a2dd6463eb00ba8ef9019dba2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d3cf0a2dd6463eb00ba8ef9019dba2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42103 } }
I20260621 07:51:12.742204 21522 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78d3cf0a2dd6463eb00ba8ef9019dba2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:51:12.742290 21522 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78d3cf0a2dd6463eb00ba8ef9019dba2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:51:12.742412 21522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78d3cf0a2dd6463eb00ba8ef9019dba2 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:12.743336 21522 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78d3cf0a2dd6463eb00ba8ef9019dba2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d3cf0a2dd6463eb00ba8ef9019dba2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42103 } }
I20260621 07:51:12.743652 21522 leader_election.cc:302] T 00000000000000000000000000000000 P 78d3cf0a2dd6463eb00ba8ef9019dba2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78d3cf0a2dd6463eb00ba8ef9019dba2; no voters:
I20260621 07:51:12.743721 21526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:12.736896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51317 (local address 127.0.0.1:42103)
0621 07:51:12.737199 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:12.737213 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:12.737254 (+ 41us) server_negotiation.cc:407] Connection header received
0621 07:51:12.737803 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:12.737826 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:12.738099 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:12.738372 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:12.738706 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:12.739560 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:12.740745 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:12.741124 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:12.741222 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:12.741992 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:12.741997 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:12.742007 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:51:12.742467 (+ 460us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:12.742541 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:51:12.742569 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:12.742650 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:12.742811 (+ 161us) server_negotiation.cc:299] Negotiation successful
0621 07:51:12.743015 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":92,"threads_started":1}
I20260621 07:51:12.743995 21522 leader_election.cc:288] T 00000000000000000000000000000000 P 78d3cf0a2dd6463eb00ba8ef9019dba2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:51:12.744092 21527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78d3cf0a2dd6463eb00ba8ef9019dba2 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:12.744285 21527 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78d3cf0a2dd6463eb00ba8ef9019dba2 [term 1 LEADER]: Becoming Leader. State: Replica: 78d3cf0a2dd6463eb00ba8ef9019dba2, State: Running, Role: LEADER
I20260621 07:51:12.744634 21527 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78d3cf0a2dd6463eb00ba8ef9019dba2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d3cf0a2dd6463eb00ba8ef9019dba2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42103 } }
I20260621 07:51:12.745070 21522 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78d3cf0a2dd6463eb00ba8ef9019dba2 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:51:12.746091 21528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d3cf0a2dd6463eb00ba8ef9019dba2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78d3cf0a2dd6463eb00ba8ef9019dba2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d3cf0a2dd6463eb00ba8ef9019dba2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42103 } } }
I20260621 07:51:12.746206 21528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d3cf0a2dd6463eb00ba8ef9019dba2 [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:12.746498 771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:51:12.746521 21529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d3cf0a2dd6463eb00ba8ef9019dba2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78d3cf0a2dd6463eb00ba8ef9019dba2. Latest consensus state: current_term: 1 leader_uuid: "78d3cf0a2dd6463eb00ba8ef9019dba2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d3cf0a2dd6463eb00ba8ef9019dba2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42103 } } }
I20260621 07:51:12.746610 21529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d3cf0a2dd6463eb00ba8ef9019dba2 [sys.catalog]: This master's current role is: LEADER
W20260621 07:51:12.749735 21543 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 78d3cf0a2dd6463eb00ba8ef9019dba2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:51:12.749810 21543 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:51:12.749891 21536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:51:12.750487 21536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:51:12.752120 21526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:12.748420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60929 (local address 127.0.0.1:42103)
0621 07:51:12.748534 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:12.748537 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:12.748715 (+ 178us) server_negotiation.cc:407] Connection header received
0621 07:51:12.748795 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:12.748808 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:12.748843 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:12.748913 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:12.749143 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:12.749783 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:12.750749 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:12.751004 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:12.751095 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:12.751765 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:12.751768 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:12.751771 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:51:12.751781 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:12.751804 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:51:12.751815 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:12.751883 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:12.751977 (+ 94us) server_negotiation.cc:299] Negotiation successful
0621 07:51:12.752021 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260621 07:51:12.755143 21536 catalog_manager.cc:1347] Generated new cluster ID: 4311ed9552864f62b5a1f6b9af17a855
I20260621 07:51:12.755194 21536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:51:12.768805 21536 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:51:12.769707 21536 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:51:12.783535 21536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78d3cf0a2dd6463eb00ba8ef9019dba2: Generated new TSK 0
I20260621 07:51:12.784150 21536 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:51:12.789194 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 21454
2026-06-21T07:51:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1253 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028028522580-771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:51:12 dist-test-slave-lzx3 krb5kdc[21553](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:51:12 dist-test-slave-lzx3 krb5kdc[21553](info): set up 2 sockets
Jun 21 07:51:12 dist-test-slave-lzx3 krb5kdc[21553](info): commencing operation
krb5kdc: starting...
W20260621 07:51:13.736850 771 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028028522580-771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028028522580-771-0/krb5kdc/test-user.keytab.
Jun 21 07:51:13 dist-test-slave-lzx3 krb5kdc[21553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:51:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:51:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 07:51:13.837433 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41995
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41995
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:13.937400 21566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:13.937620 21566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:13.937682 21566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:13.941186 21566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:51:13.941257 21566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:13.941293 21566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:13.941347 21566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:51:13.941386 21566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:51:13.946233 21566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:13.947053 21566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:13.947885 21566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:13.955279 21574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:13.955260 21571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:13.955260 21572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:13.955412 21566 server_base.cc:1034] running on GCE node
I20260621 07:51:13.956184 21566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:13.957158 21566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:13.958339 21566 hybrid_clock.cc:648] HybridClock initialized: now 1782028273958334 us; error 36 us; skew 500 ppm
Jun 21 07:51:13 dist-test-slave-lzx3 krb5kdc[21553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:13.960621 21566 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 07:51:13.961313 21566 webserver.cc:460] Webserver started at http://127.0.0.1:40855/ using document root <none> and password file <none>
I20260621 07:51:13.961773 21566 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:13.961830 21566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:13.961982 21566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:13.963671 21566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "5019f1135e7846628d2f4f447b175ef1"
format_stamp: "Formatted at 2026-06-21 07:51:13 on dist-test-slave-lzx3"
server_key: "61fa577395b1b9c8465cae6825136459"
server_key_iv: "9f67766005c336239bf99e25671cf685"
server_key_version: "encryptionkey@0"
I20260621 07:51:13.964143 21566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "5019f1135e7846628d2f4f447b175ef1"
format_stamp: "Formatted at 2026-06-21 07:51:13 on dist-test-slave-lzx3"
server_key: "61fa577395b1b9c8465cae6825136459"
server_key_iv: "9f67766005c336239bf99e25671cf685"
server_key_version: "encryptionkey@0"
I20260621 07:51:13.967093 21566 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 07:51:13.969044 21581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:13.970002 21566 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:51:13.970113 21566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "5019f1135e7846628d2f4f447b175ef1"
format_stamp: "Formatted at 2026-06-21 07:51:13 on dist-test-slave-lzx3"
server_key: "61fa577395b1b9c8465cae6825136459"
server_key_iv: "9f67766005c336239bf99e25671cf685"
server_key_version: "encryptionkey@0"
I20260621 07:51:13.970197 21566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:13.974025 21566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:13.976215 21566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:13.976369 21566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:13.982183 21566 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41995
I20260621 07:51:13.982247 21633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41995 every 8 connection(s)
I20260621 07:51:13.982983 21566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:51:13.985291 21634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:13.989549 21634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5019f1135e7846628d2f4f447b175ef1: Bootstrap starting.
I20260621 07:51:13.991657 21634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5019f1135e7846628d2f4f447b175ef1: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:13.991667 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 21566
I20260621 07:51:13.991776 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:51:13.992036 771 external_mini_cluster.cc:1428] Setting key 4bd07d59bf9b93e26c7684420f394e73
I20260621 07:51:13.992383 21634 log.cc:824] T 00000000000000000000000000000000 P 5019f1135e7846628d2f4f447b175ef1: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:13.994280 21634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5019f1135e7846628d2f4f447b175ef1: No bootstrap required, opened a new log
I20260621 07:51:13.996246 21638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:13.993014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54761 (local address 127.0.0.1:41995)
0621 07:51:13.993396 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:13.993421 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:13.993460 (+ 39us) server_negotiation.cc:407] Connection header received
0621 07:51:13.994014 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:13.994027 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:13.994248 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:13.994550 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:13.995234 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:13.995238 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:13.995247 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:51:13.995256 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:13.995295 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:51:13.995310 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:13.995409 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:13.995521 (+ 112us) server_negotiation.cc:299] Negotiation successful
0621 07:51:13.995664 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":127,"threads_started":1}
I20260621 07:51:13.997850 21634 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5019f1135e7846628d2f4f447b175ef1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5019f1135e7846628d2f4f447b175ef1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41995 } }
I20260621 07:51:13.998102 21634 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5019f1135e7846628d2f4f447b175ef1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:13.998191 21634 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5019f1135e7846628d2f4f447b175ef1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5019f1135e7846628d2f4f447b175ef1, State: Initialized, Role: FOLLOWER
I20260621 07:51:13.998526 771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:51:13.998638 21634 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5019f1135e7846628d2f4f447b175ef1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5019f1135e7846628d2f4f447b175ef1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41995 } }
I20260621 07:51:13.998782 21634 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5019f1135e7846628d2f4f447b175ef1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:51:13.998840 21634 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5019f1135e7846628d2f4f447b175ef1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:51:13.998924 21634 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5019f1135e7846628d2f4f447b175ef1 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:13.999662 21634 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5019f1135e7846628d2f4f447b175ef1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5019f1135e7846628d2f4f447b175ef1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41995 } }
I20260621 07:51:13.999974 21634 leader_election.cc:302] T 00000000000000000000000000000000 P 5019f1135e7846628d2f4f447b175ef1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5019f1135e7846628d2f4f447b175ef1; no voters:
I20260621 07:51:14.000264 21634 leader_election.cc:288] T 00000000000000000000000000000000 P 5019f1135e7846628d2f4f447b175ef1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:51:14.000350 21643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5019f1135e7846628d2f4f447b175ef1 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:14.000648 21643 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5019f1135e7846628d2f4f447b175ef1 [term 1 LEADER]: Becoming Leader. State: Replica: 5019f1135e7846628d2f4f447b175ef1, State: Running, Role: LEADER
I20260621 07:51:14.000967 21643 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5019f1135e7846628d2f4f447b175ef1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5019f1135e7846628d2f4f447b175ef1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41995 } }
I20260621 07:51:14.001428 21634 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5019f1135e7846628d2f4f447b175ef1 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:51:14.002494 21645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5019f1135e7846628d2f4f447b175ef1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5019f1135e7846628d2f4f447b175ef1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5019f1135e7846628d2f4f447b175ef1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41995 } } }
I20260621 07:51:14.002619 21645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5019f1135e7846628d2f4f447b175ef1 [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:14.003007 21653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:51:14.003580 21646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5019f1135e7846628d2f4f447b175ef1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5019f1135e7846628d2f4f447b175ef1. Latest consensus state: current_term: 1 leader_uuid: "5019f1135e7846628d2f4f447b175ef1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5019f1135e7846628d2f4f447b175ef1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41995 } } }
I20260621 07:51:14.003746 21646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5019f1135e7846628d2f4f447b175ef1 [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:14.004920 21638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:14.000391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48179 (local address 127.0.0.1:41995)
0621 07:51:14.000494 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:14.000497 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:14.003668 (+ 3171us) server_negotiation.cc:407] Connection header received
0621 07:51:14.003796 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:14.003799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:14.003829 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:14.003910 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:14.004530 (+ 620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:14.004533 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:14.004536 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:51:14.004547 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:14.004597 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:51:14.004609 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:14.004701 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:14.004796 (+ 95us) server_negotiation.cc:299] Negotiation successful
0621 07:51:14.004826 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260621 07:51:14.005934 21653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:51:14.010582 21653 catalog_manager.cc:1347] Generated new cluster ID: c540e341fb404ca385d0be42584a7da2
I20260621 07:51:14.010641 21653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:51:14.021837 21653 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:51:14.022110 21653 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:51:14.033013 21653 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5019f1135e7846628d2f4f447b175ef1: Generated new TSK 0
I20260621 07:51:14.033677 21653 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:51:14.041180 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 21566
2026-06-21T07:51:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1252 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028028522580-771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:51:14 dist-test-slave-lzx3 krb5kdc[21665](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:51:14 dist-test-slave-lzx3 krb5kdc[21665](info): set up 2 sockets
Jun 21 07:51:14 dist-test-slave-lzx3 krb5kdc[21665](info): commencing operation
krb5kdc: starting...
W20260621 07:51:14.964793 771 mini_kdc.cc:121] Time spent starting KDC: real 0.905s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028028522580-771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028028522580-771-0/krb5kdc/test-user.keytab.
Jun 21 07:51:15 dist-test-slave-lzx3 krb5kdc[21665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:51:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:51:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 07:51:15.066898 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40535
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:15.160599 21678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:15.160857 21678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:15.160945 21678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:15.164808 21678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:51:15.164884 21678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:15.164921 21678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:15.164974 21678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:51:15.165010 21678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:51:15.169735 21678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:15.170616 21678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:15.171516 21678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:15.179328 21683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:15.179271 21686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:15.179455 21684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:15.179477 21678 server_base.cc:1034] running on GCE node
I20260621 07:51:15.180053 21678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:15.181120 21678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:15.182274 21678 hybrid_clock.cc:648] HybridClock initialized: now 1782028275182256 us; error 29 us; skew 500 ppm
Jun 21 07:51:15 dist-test-slave-lzx3 krb5kdc[21665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:15.187119 21678 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 07:51:15.187784 21678 webserver.cc:460] Webserver started at http://127.0.0.1:37249/ using document root <none> and password file <none>
I20260621 07:51:15.188189 21678 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:15.188237 21678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:15.188367 21678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:15.190039 21678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "df5d90ede52547c6a51de02f15e2671a"
format_stamp: "Formatted at 2026-06-21 07:51:15 on dist-test-slave-lzx3"
server_key: "b09f897e7e396a64b8ffbcb013d8faeb"
server_key_iv: "f0e3a1789b5deba22444228214b917ba"
server_key_version: "encryptionkey@0"
I20260621 07:51:15.190474 21678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "df5d90ede52547c6a51de02f15e2671a"
format_stamp: "Formatted at 2026-06-21 07:51:15 on dist-test-slave-lzx3"
server_key: "b09f897e7e396a64b8ffbcb013d8faeb"
server_key_iv: "f0e3a1789b5deba22444228214b917ba"
server_key_version: "encryptionkey@0"
I20260621 07:51:15.193547 21678 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:51:15.195509 21693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:15.196471 21678 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:51:15.196614 21678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "df5d90ede52547c6a51de02f15e2671a"
format_stamp: "Formatted at 2026-06-21 07:51:15 on dist-test-slave-lzx3"
server_key: "b09f897e7e396a64b8ffbcb013d8faeb"
server_key_iv: "f0e3a1789b5deba22444228214b917ba"
server_key_version: "encryptionkey@0"
I20260621 07:51:15.196693 21678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:15.213644 21678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:15.215943 21678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:15.216084 21678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:15.222159 21678 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40535
I20260621 07:51:15.222172 21745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40535 every 8 connection(s)
I20260621 07:51:15.222991 21678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:51:15.225203 21746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:15.229499 21746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df5d90ede52547c6a51de02f15e2671a: Bootstrap starting.
I20260621 07:51:15.231388 21746 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df5d90ede52547c6a51de02f15e2671a: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:15.231415 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 21678
I20260621 07:51:15.231544 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:51:15.231789 771 external_mini_cluster.cc:1428] Setting key 9ab5a3545413404e92d5969a39f2d0c1
I20260621 07:51:15.232069 21746 log.cc:824] T 00000000000000000000000000000000 P df5d90ede52547c6a51de02f15e2671a: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:15.233826 21746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df5d90ede52547c6a51de02f15e2671a: No bootstrap required, opened a new log
I20260621 07:51:15.236780 21746 raft_consensus.cc:348] T 00000000000000000000000000000000 P df5d90ede52547c6a51de02f15e2671a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5d90ede52547c6a51de02f15e2671a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40535 } }
I20260621 07:51:15.236981 21746 raft_consensus.cc:374] T 00000000000000000000000000000000 P df5d90ede52547c6a51de02f15e2671a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:15.237078 21746 raft_consensus.cc:729] T 00000000000000000000000000000000 P df5d90ede52547c6a51de02f15e2671a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df5d90ede52547c6a51de02f15e2671a, State: Initialized, Role: FOLLOWER
I20260621 07:51:15.237499 21746 consensus_queue.cc:260] T 00000000000000000000000000000000 P df5d90ede52547c6a51de02f15e2671a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5d90ede52547c6a51de02f15e2671a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40535 } }
I20260621 07:51:15.237670 21746 raft_consensus.cc:388] T 00000000000000000000000000000000 P df5d90ede52547c6a51de02f15e2671a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:51:15.237752 21746 raft_consensus.cc:482] T 00000000000000000000000000000000 P df5d90ede52547c6a51de02f15e2671a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:51:15.237850 21746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df5d90ede52547c6a51de02f15e2671a [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:15.238993 21746 raft_consensus.cc:504] T 00000000000000000000000000000000 P df5d90ede52547c6a51de02f15e2671a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5d90ede52547c6a51de02f15e2671a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40535 } }
I20260621 07:51:15.239300 21746 leader_election.cc:302] T 00000000000000000000000000000000 P df5d90ede52547c6a51de02f15e2671a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df5d90ede52547c6a51de02f15e2671a; no voters:
Jun 21 07:51:15 dist-test-slave-lzx3 krb5kdc[21665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260621 07:51:15.239583 21746 leader_election.cc:288] T 00000000000000000000000000000000 P df5d90ede52547c6a51de02f15e2671a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:51:15.239694 21751 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df5d90ede52547c6a51de02f15e2671a [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:15.239926 21751 raft_consensus.cc:686] T 00000000000000000000000000000000 P df5d90ede52547c6a51de02f15e2671a [term 1 LEADER]: Becoming Leader. State: Replica: df5d90ede52547c6a51de02f15e2671a, State: Running, Role: LEADER
I20260621 07:51:15.240307 21751 consensus_queue.cc:237] T 00000000000000000000000000000000 P df5d90ede52547c6a51de02f15e2671a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5d90ede52547c6a51de02f15e2671a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40535 } }
I20260621 07:51:15.240798 21746 sys_catalog.cc:564] T 00000000000000000000000000000000 P df5d90ede52547c6a51de02f15e2671a [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:51:15.241492 21753 sys_catalog.cc:455] T 00000000000000000000000000000000 P df5d90ede52547c6a51de02f15e2671a [sys.catalog]: SysCatalogTable state changed. Reason: New leader df5d90ede52547c6a51de02f15e2671a. Latest consensus state: current_term: 1 leader_uuid: "df5d90ede52547c6a51de02f15e2671a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5d90ede52547c6a51de02f15e2671a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40535 } } }
I20260621 07:51:15.241689 21753 sys_catalog.cc:458] T 00000000000000000000000000000000 P df5d90ede52547c6a51de02f15e2671a [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:15.241919 21752 sys_catalog.cc:455] T 00000000000000000000000000000000 P df5d90ede52547c6a51de02f15e2671a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df5d90ede52547c6a51de02f15e2671a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5d90ede52547c6a51de02f15e2671a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40535 } } }
I20260621 07:51:15.242040 21752 sys_catalog.cc:458] T 00000000000000000000000000000000 P df5d90ede52547c6a51de02f15e2671a [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:15.242156 21760 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:51:15.245539 21750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:15.232745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:40129 (local address 127.0.0.1:40535)
0621 07:51:15.233150 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:15.233163 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:15.233194 (+ 31us) server_negotiation.cc:407] Connection header received
0621 07:51:15.233756 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:15.233770 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:15.234187 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:15.234479 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:15.235194 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:15.236027 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:15.236870 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:15.237441 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:15.237605 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:15.240323 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:15.240329 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:15.240343 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:15.240369 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:15.242439 (+ 2070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:15.242969 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:15.242975 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:15.242981 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:15.243103 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:15.243503 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:15.243506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:15.243508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:15.243920 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:15.244132 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:15.244633 (+ 501us) server_negotiation.cc:299] Negotiation successful
0621 07:51:15.244821 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":117,"threads_started":1}
I20260621 07:51:15.245616 21760 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:51:15.248206 771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:51:15.250417 21760 catalog_manager.cc:1347] Generated new cluster ID: cbbe94c646734a0e96524d1beeb5c9c7
I20260621 07:51:15.250485 21760 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:51:15.263175 21750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:15.255536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:37815 (local address 127.0.0.1:40535)
0621 07:51:15.255644 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:15.255648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:15.255659 (+ 11us) server_negotiation.cc:407] Connection header received
0621 07:51:15.255687 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:15.255689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:15.255723 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:15.255791 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:15.256455 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:15.257096 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:15.258122 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:15.258463 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:15.258552 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:15.259999 (+ 1447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:15.260002 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:15.260004 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:15.260024 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:15.261548 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:15.262029 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:15.262032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:15.262033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:15.262097 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:15.262400 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:15.262403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:15.262404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:15.262568 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:15.262688 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:15.262957 (+ 269us) server_negotiation.cc:299] Negotiation successful
0621 07:51:15.263036 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 07:51:15.273664 21760 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:51:15.274372 21760 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:51:15.280624 21760 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df5d90ede52547c6a51de02f15e2671a: Generated new TSK 0
I20260621 07:51:15.281203 21760 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:51:15.290525 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 21678
W20260621 07:51:15.295457 21663 connection.cc:535] client connection to 127.0.0.1:40535 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40535): Connection reset by peer (error 104)
2026-06-21T07:51:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1249 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028028522580-771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:51:15 dist-test-slave-lzx3 krb5kdc[21777](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:51:15 dist-test-slave-lzx3 krb5kdc[21777](info): set up 2 sockets
Jun 21 07:51:15 dist-test-slave-lzx3 krb5kdc[21777](info): commencing operation
krb5kdc: starting...
W20260621 07:51:16.272171 771 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028028522580-771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028028522580-771-0/krb5kdc/test-user.keytab.
Jun 21 07:51:16 dist-test-slave-lzx3 krb5kdc[21777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:51:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:51:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 07:51:16.373189 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39975
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:16.475003 21790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:16.475267 21790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:16.475337 21790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:16.479063 21790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:51:16.479147 21790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:16.479192 21790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:16.479249 21790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:51:16.479283 21790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:51:16.484299 21790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:16.485208 21790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:16.486132 21790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:16.493885 21796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:16.493986 21790 server_base.cc:1034] running on GCE node
W20260621 07:51:16.493918 21798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:16.493898 21795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:16.494618 21790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:16.495504 21790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:16.496630 21790 hybrid_clock.cc:648] HybridClock initialized: now 1782028276496613 us; error 28 us; skew 500 ppm
Jun 21 07:51:16 dist-test-slave-lzx3 krb5kdc[21777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028276, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:16.498936 21790 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 07:51:16.499742 21790 webserver.cc:460] Webserver started at http://127.0.0.1:42179/ using document root <none> and password file <none>
I20260621 07:51:16.500283 21790 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:16.500356 21790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:16.500674 21790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:16.502539 21790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "a6c32ba57a874079a237d7a1e475fc30"
format_stamp: "Formatted at 2026-06-21 07:51:16 on dist-test-slave-lzx3"
server_key: "fa821aa03e40cdc0a37d7d7d6e60835e"
server_key_iv: "06f6a50ccad98dd3528a2114ac01f7ab"
server_key_version: "encryptionkey@0"
I20260621 07:51:16.503095 21790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "a6c32ba57a874079a237d7a1e475fc30"
format_stamp: "Formatted at 2026-06-21 07:51:16 on dist-test-slave-lzx3"
server_key: "fa821aa03e40cdc0a37d7d7d6e60835e"
server_key_iv: "06f6a50ccad98dd3528a2114ac01f7ab"
server_key_version: "encryptionkey@0"
I20260621 07:51:16.506423 21790 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 07:51:16.508548 21805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:16.509719 21790 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:51:16.509836 21790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "a6c32ba57a874079a237d7a1e475fc30"
format_stamp: "Formatted at 2026-06-21 07:51:16 on dist-test-slave-lzx3"
server_key: "fa821aa03e40cdc0a37d7d7d6e60835e"
server_key_iv: "06f6a50ccad98dd3528a2114ac01f7ab"
server_key_version: "encryptionkey@0"
I20260621 07:51:16.509927 21790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:16.557888 21790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:16.560492 21790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:16.560760 21790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:16.566951 21790 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39975
I20260621 07:51:16.566962 21857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39975 every 8 connection(s)
I20260621 07:51:16.567747 21790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:51:16.568310 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 21790
I20260621 07:51:16.568436 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:51:16.568743 771 external_mini_cluster.cc:1428] Setting key d0a8308a146ae7ea89575757444aa974
I20260621 07:51:16.570590 21858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:16.575627 21858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6c32ba57a874079a237d7a1e475fc30: Bootstrap starting.
I20260621 07:51:16.576753 21861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:16.569944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:57933 (local address 127.0.0.1:39975)
0621 07:51:16.570347 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:16.570360 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:16.570391 (+ 31us) server_negotiation.cc:407] Connection header received
0621 07:51:16.571010 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:16.571027 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:16.571342 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:16.571660 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:16.572061 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:16.573094 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:16.573903 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:16.574321 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:16.574483 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:16.575281 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:16.575285 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:16.575296 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:51:16.575627 (+ 331us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:16.575693 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:51:16.575723 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:16.575850 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:16.576005 (+ 155us) server_negotiation.cc:299] Negotiation successful
0621 07:51:16.576119 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":128,"threads_started":1}
I20260621 07:51:16.577692 21858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6c32ba57a874079a237d7a1e475fc30: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:16.578454 21858 log.cc:824] T 00000000000000000000000000000000 P a6c32ba57a874079a237d7a1e475fc30: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:16.579366 771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:51:16.580217 21858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6c32ba57a874079a237d7a1e475fc30: No bootstrap required, opened a new log
I20260621 07:51:16.583591 21858 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6c32ba57a874079a237d7a1e475fc30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c32ba57a874079a237d7a1e475fc30" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39975 } }
I20260621 07:51:16.583765 21858 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6c32ba57a874079a237d7a1e475fc30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:16.583837 21858 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6c32ba57a874079a237d7a1e475fc30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6c32ba57a874079a237d7a1e475fc30, State: Initialized, Role: FOLLOWER
I20260621 07:51:16.584215 21858 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6c32ba57a874079a237d7a1e475fc30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c32ba57a874079a237d7a1e475fc30" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39975 } }
I20260621 07:51:16.584347 21858 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6c32ba57a874079a237d7a1e475fc30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:51:16.584403 21858 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6c32ba57a874079a237d7a1e475fc30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:51:16.584481 21858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6c32ba57a874079a237d7a1e475fc30 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:16.585217 21858 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6c32ba57a874079a237d7a1e475fc30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c32ba57a874079a237d7a1e475fc30" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39975 } }
I20260621 07:51:16.585523 21858 leader_election.cc:302] T 00000000000000000000000000000000 P a6c32ba57a874079a237d7a1e475fc30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6c32ba57a874079a237d7a1e475fc30; no voters:
I20260621 07:51:16.585830 21858 leader_election.cc:288] T 00000000000000000000000000000000 P a6c32ba57a874079a237d7a1e475fc30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:51:16.585888 21868 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6c32ba57a874079a237d7a1e475fc30 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:16.586084 21868 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6c32ba57a874079a237d7a1e475fc30 [term 1 LEADER]: Becoming Leader. State: Replica: a6c32ba57a874079a237d7a1e475fc30, State: Running, Role: LEADER
I20260621 07:51:16.586385 21868 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6c32ba57a874079a237d7a1e475fc30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c32ba57a874079a237d7a1e475fc30" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39975 } }
I20260621 07:51:16.586944 21858 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6c32ba57a874079a237d7a1e475fc30 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:51:16.587368 21861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:16.581281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:38795 (local address 127.0.0.1:39975)
0621 07:51:16.581367 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:16.581370 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:16.584000 (+ 2630us) server_negotiation.cc:407] Connection header received
0621 07:51:16.584137 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:16.584140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:16.584178 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:16.584250 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:16.584454 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:16.584938 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:16.585782 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:16.586123 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:16.586211 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:16.586971 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:16.586973 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:16.586976 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:51:16.586985 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:16.587008 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:51:16.587021 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:16.587112 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:16.587207 (+ 95us) server_negotiation.cc:299] Negotiation successful
0621 07:51:16.587234 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260621 07:51:16.588022 21870 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6c32ba57a874079a237d7a1e475fc30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6c32ba57a874079a237d7a1e475fc30. Latest consensus state: current_term: 1 leader_uuid: "a6c32ba57a874079a237d7a1e475fc30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c32ba57a874079a237d7a1e475fc30" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39975 } } }
I20260621 07:51:16.588135 21870 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6c32ba57a874079a237d7a1e475fc30 [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:16.588317 21869 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6c32ba57a874079a237d7a1e475fc30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6c32ba57a874079a237d7a1e475fc30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c32ba57a874079a237d7a1e475fc30" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39975 } } }
I20260621 07:51:16.588413 21869 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6c32ba57a874079a237d7a1e475fc30 [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:16.588500 21879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:51:16.591223 21879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:51:16.596132 21879 catalog_manager.cc:1347] Generated new cluster ID: 0b30456b75fb41548e64ae76ce01b066
I20260621 07:51:16.596196 21879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:51:16.611495 21879 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:51:16.612264 21879 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:51:16.621337 21879 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6c32ba57a874079a237d7a1e475fc30: Generated new TSK 0
I20260621 07:51:16.621865 21879 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:51:16.629072 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 21790
W20260621 07:51:16.634238 21775 connection.cc:535] client connection to 127.0.0.1:39975 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39975): Connection reset by peer (error 104)
2026-06-21T07:51:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1338 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028028522580-771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:51:16 dist-test-slave-lzx3 krb5kdc[21889](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:51:16 dist-test-slave-lzx3 krb5kdc[21889](info): set up 2 sockets
Jun 21 07:51:16 dist-test-slave-lzx3 krb5kdc[21889](info): commencing operation
krb5kdc: starting...
W20260621 07:51:17.572772 771 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028028522580-771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028028522580-771-0/krb5kdc/test-user.keytab.
Jun 21 07:51:17 dist-test-slave-lzx3 krb5kdc[21889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:51:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:51:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 07:51:17.679910 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36735
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:17.777509 21902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:17.777693 21902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:17.777751 21902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:17.781608 21902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:51:17.781656 21902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:17.781675 21902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:17.781692 21902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:51:17.781744 21902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:51:17.786305 21902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36735
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:17.787050 21902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:17.787855 21902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:17.795042 21910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:17.795061 21907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:17.795104 21902 server_base.cc:1034] running on GCE node
W20260621 07:51:17.795061 21908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:17.795694 21902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:17.796515 21902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:17.797701 21902 hybrid_clock.cc:648] HybridClock initialized: now 1782028277797681 us; error 31 us; skew 500 ppm
Jun 21 07:51:17 dist-test-slave-lzx3 krb5kdc[21889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028277, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:17.800060 21902 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 07:51:17.800918 21902 webserver.cc:460] Webserver started at http://127.0.0.1:46777/ using document root <none> and password file <none>
I20260621 07:51:17.801357 21902 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:17.801411 21902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:17.801563 21902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:17.803290 21902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "6a011b80bd664dd0905d752dbf44b118"
format_stamp: "Formatted at 2026-06-21 07:51:17 on dist-test-slave-lzx3"
server_key: "7275bf191ab10c6cb8b5b420565f9074"
server_key_iv: "aacf7a98346eaf4975b1681335364ce9"
server_key_version: "encryptionkey@0"
I20260621 07:51:17.803759 21902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "6a011b80bd664dd0905d752dbf44b118"
format_stamp: "Formatted at 2026-06-21 07:51:17 on dist-test-slave-lzx3"
server_key: "7275bf191ab10c6cb8b5b420565f9074"
server_key_iv: "aacf7a98346eaf4975b1681335364ce9"
server_key_version: "encryptionkey@0"
I20260621 07:51:17.806785 21902 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 07:51:17.808753 21917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:17.809751 21902 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:51:17.809864 21902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "6a011b80bd664dd0905d752dbf44b118"
format_stamp: "Formatted at 2026-06-21 07:51:17 on dist-test-slave-lzx3"
server_key: "7275bf191ab10c6cb8b5b420565f9074"
server_key_iv: "aacf7a98346eaf4975b1681335364ce9"
server_key_version: "encryptionkey@0"
I20260621 07:51:17.809953 21902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:17.813954 21902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:17.816587 21902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:17.816776 21902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:17.822755 21902 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36735
I20260621 07:51:17.822770 21969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36735 every 8 connection(s)
I20260621 07:51:17.823454 21902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:51:17.824369 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 21902
I20260621 07:51:17.824497 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:51:17.824770 771 external_mini_cluster.cc:1428] Setting key 585f9533309b2646929f9e0a7c75ba5e
I20260621 07:51:17.826090 21970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:17.828914 21973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:17.825847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:34079 (local address 127.0.0.1:36735)
0621 07:51:17.826165 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:17.826183 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:17.826221 (+ 38us) server_negotiation.cc:407] Connection header received
0621 07:51:17.826716 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:17.826729 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:17.826947 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:17.827176 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:17.827834 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:17.827838 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:17.827847 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:51:17.827855 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:17.827895 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:51:17.827910 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:17.828003 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:17.828109 (+ 106us) server_negotiation.cc:299] Negotiation successful
0621 07:51:17.828246 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":111,"threads_started":1}
I20260621 07:51:17.831447 771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:51:17.831507 21970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a011b80bd664dd0905d752dbf44b118: Bootstrap starting.
I20260621 07:51:17.834060 21970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a011b80bd664dd0905d752dbf44b118: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:17.834767 21970 log.cc:824] T 00000000000000000000000000000000 P 6a011b80bd664dd0905d752dbf44b118: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:17.835161 21973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:17.833564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:42943 (local address 127.0.0.1:36735)
0621 07:51:17.833680 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:17.833684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:17.833868 (+ 184us) server_negotiation.cc:407] Connection header received
0621 07:51:17.833940 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:17.833946 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:17.833986 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:17.834087 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:17.834742 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:17.834746 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:17.834748 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:51:17.834759 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:17.834786 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:51:17.834798 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:17.834897 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:17.834990 (+ 93us) server_negotiation.cc:299] Negotiation successful
0621 07:51:17.835033 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 07:51:17.836601 21970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a011b80bd664dd0905d752dbf44b118: No bootstrap required, opened a new log
I20260621 07:51:17.839625 21970 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a011b80bd664dd0905d752dbf44b118 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a011b80bd664dd0905d752dbf44b118" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36735 } }
I20260621 07:51:17.839850 21970 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a011b80bd664dd0905d752dbf44b118 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:17.839927 21970 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a011b80bd664dd0905d752dbf44b118 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a011b80bd664dd0905d752dbf44b118, State: Initialized, Role: FOLLOWER
I20260621 07:51:17.840320 21970 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a011b80bd664dd0905d752dbf44b118 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a011b80bd664dd0905d752dbf44b118" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36735 } }
I20260621 07:51:17.840440 21970 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a011b80bd664dd0905d752dbf44b118 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:51:17.840534 21970 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a011b80bd664dd0905d752dbf44b118 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:51:17.840647 21970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a011b80bd664dd0905d752dbf44b118 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:17.841395 21970 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a011b80bd664dd0905d752dbf44b118 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a011b80bd664dd0905d752dbf44b118" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36735 } }
I20260621 07:51:17.841730 21970 leader_election.cc:302] T 00000000000000000000000000000000 P 6a011b80bd664dd0905d752dbf44b118 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a011b80bd664dd0905d752dbf44b118; no voters:
I20260621 07:51:17.842020 21970 leader_election.cc:288] T 00000000000000000000000000000000 P 6a011b80bd664dd0905d752dbf44b118 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:51:17.842135 21980 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a011b80bd664dd0905d752dbf44b118 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:17.842374 21980 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a011b80bd664dd0905d752dbf44b118 [term 1 LEADER]: Becoming Leader. State: Replica: 6a011b80bd664dd0905d752dbf44b118, State: Running, Role: LEADER
I20260621 07:51:17.842669 21980 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a011b80bd664dd0905d752dbf44b118 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a011b80bd664dd0905d752dbf44b118" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36735 } }
I20260621 07:51:17.843055 21970 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a011b80bd664dd0905d752dbf44b118 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:51:17.844087 21981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a011b80bd664dd0905d752dbf44b118 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a011b80bd664dd0905d752dbf44b118" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a011b80bd664dd0905d752dbf44b118" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36735 } } }
I20260621 07:51:17.844197 21981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a011b80bd664dd0905d752dbf44b118 [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:17.844501 21982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a011b80bd664dd0905d752dbf44b118 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a011b80bd664dd0905d752dbf44b118. Latest consensus state: current_term: 1 leader_uuid: "6a011b80bd664dd0905d752dbf44b118" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a011b80bd664dd0905d752dbf44b118" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36735 } } }
I20260621 07:51:17.844609 21982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a011b80bd664dd0905d752dbf44b118 [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:17.844996 21989 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:51:17.847309 21989 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:51:17.851971 21989 catalog_manager.cc:1347] Generated new cluster ID: 95b34e966a2343e8b00b626d0cf32c3a
I20260621 07:51:17.852041 21989 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:51:17.859699 21989 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:51:17.859951 21989 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:51:17.866753 21989 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a011b80bd664dd0905d752dbf44b118: Generated new TSK 0
I20260621 07:51:17.867264 21989 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:51:17.869607 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 21902
2026-06-21T07:51:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1240 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028028522580-771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:51:17 dist-test-slave-lzx3 krb5kdc[22001](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:51:17 dist-test-slave-lzx3 krb5kdc[22001](info): set up 2 sockets
Jun 21 07:51:17 dist-test-slave-lzx3 krb5kdc[22001](info): commencing operation
krb5kdc: starting...
W20260621 07:51:18.804445 771 mini_kdc.cc:121] Time spent starting KDC: real 0.916s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028028522580-771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028028522580-771-0/krb5kdc/test-user.keytab.
Jun 21 07:51:18 dist-test-slave-lzx3 krb5kdc[22001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:51:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:51:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 07:51:18.905406 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41835
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:19.003306 22014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:19.003530 22014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:19.003569 22014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:19.007429 22014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:51:19.007476 22014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:19.007494 22014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:19.007545 22014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:51:19.007570 22014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:51:19.012284 22014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41835
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:19.013137 22014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:19.014038 22014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:19.021636 22020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:19.021639 22022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:19.021792 22014 server_base.cc:1034] running on GCE node
W20260621 07:51:19.021696 22019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:19.022437 22014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:19.023329 22014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:19.024501 22014 hybrid_clock.cc:648] HybridClock initialized: now 1782028279024476 us; error 37 us; skew 500 ppm
Jun 21 07:51:19 dist-test-slave-lzx3 krb5kdc[22001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:19.026986 22014 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 07:51:19.027715 22014 webserver.cc:460] Webserver started at http://127.0.0.1:46317/ using document root <none> and password file <none>
I20260621 07:51:19.028162 22014 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:19.028213 22014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:19.028352 22014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:19.030205 22014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "751bf0e1893444738afe00d496b8b5c9"
format_stamp: "Formatted at 2026-06-21 07:51:19 on dist-test-slave-lzx3"
server_key: "821621146dea8678c28b82e4e6c0cce0"
server_key_iv: "15c45d841bfa616a472230e77ff0bde4"
server_key_version: "encryptionkey@0"
I20260621 07:51:19.030647 22014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "751bf0e1893444738afe00d496b8b5c9"
format_stamp: "Formatted at 2026-06-21 07:51:19 on dist-test-slave-lzx3"
server_key: "821621146dea8678c28b82e4e6c0cce0"
server_key_iv: "15c45d841bfa616a472230e77ff0bde4"
server_key_version: "encryptionkey@0"
I20260621 07:51:19.033910 22014 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260621 07:51:19.035892 22029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:19.037000 22014 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:51:19.037114 22014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "751bf0e1893444738afe00d496b8b5c9"
format_stamp: "Formatted at 2026-06-21 07:51:19 on dist-test-slave-lzx3"
server_key: "821621146dea8678c28b82e4e6c0cce0"
server_key_iv: "15c45d841bfa616a472230e77ff0bde4"
server_key_version: "encryptionkey@0"
I20260621 07:51:19.037204 22014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:19.047889 22014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:19.050226 22014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:19.050400 22014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:19.056361 22014 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41835
I20260621 07:51:19.056423 22081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41835 every 8 connection(s)
I20260621 07:51:19.057155 22014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:51:19.059492 22082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:19.059963 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 22014
I20260621 07:51:19.060091 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:51:19.060334 771 external_mini_cluster.cc:1428] Setting key a83c0b3e47c0ac52e8a1a8cecceae6ca
I20260621 07:51:19.065662 22082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 751bf0e1893444738afe00d496b8b5c9: Bootstrap starting.
I20260621 07:51:19.068058 22082 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 751bf0e1893444738afe00d496b8b5c9: Neither blocks nor log segments found. Creating new log.
Jun 21 07:51:19 dist-test-slave-lzx3 krb5kdc[22001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260621 07:51:19.068790 22082 log.cc:824] T 00000000000000000000000000000000 P 751bf0e1893444738afe00d496b8b5c9: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:19.070297 22082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 751bf0e1893444738afe00d496b8b5c9: No bootstrap required, opened a new log
I20260621 07:51:19.073158 22085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:19.061627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:52017 (local address 127.0.0.1:41835)
0621 07:51:19.062002 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:19.062014 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:19.062045 (+ 31us) server_negotiation.cc:407] Connection header received
0621 07:51:19.062634 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:19.062648 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:19.062928 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:19.063179 (+ 251us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:19.064040 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:19.064947 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:19.065982 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:19.066370 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:19.066471 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:19.069045 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:19.069049 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:19.069059 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:19.069081 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:19.070576 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:19.071085 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:19.071090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:19.071094 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:19.071189 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:19.071518 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:19.071522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:19.071523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:19.071845 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:19.072091 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:19.072363 (+ 272us) server_negotiation.cc:299] Negotiation successful
0621 07:51:19.072543 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":118,"threads_started":1}
I20260621 07:51:19.073558 22082 raft_consensus.cc:348] T 00000000000000000000000000000000 P 751bf0e1893444738afe00d496b8b5c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751bf0e1893444738afe00d496b8b5c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41835 } }
I20260621 07:51:19.073765 22082 raft_consensus.cc:374] T 00000000000000000000000000000000 P 751bf0e1893444738afe00d496b8b5c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:19.073843 22082 raft_consensus.cc:729] T 00000000000000000000000000000000 P 751bf0e1893444738afe00d496b8b5c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 751bf0e1893444738afe00d496b8b5c9, State: Initialized, Role: FOLLOWER
I20260621 07:51:19.074281 22082 consensus_queue.cc:260] T 00000000000000000000000000000000 P 751bf0e1893444738afe00d496b8b5c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751bf0e1893444738afe00d496b8b5c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41835 } }
I20260621 07:51:19.074396 22082 raft_consensus.cc:388] T 00000000000000000000000000000000 P 751bf0e1893444738afe00d496b8b5c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:51:19.074474 22082 raft_consensus.cc:482] T 00000000000000000000000000000000 P 751bf0e1893444738afe00d496b8b5c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:51:19.074586 22082 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 751bf0e1893444738afe00d496b8b5c9 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:19.075565 22082 raft_consensus.cc:504] T 00000000000000000000000000000000 P 751bf0e1893444738afe00d496b8b5c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751bf0e1893444738afe00d496b8b5c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41835 } }
I20260621 07:51:19.075771 771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:51:19.075907 22082 leader_election.cc:302] T 00000000000000000000000000000000 P 751bf0e1893444738afe00d496b8b5c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 751bf0e1893444738afe00d496b8b5c9; no voters:
I20260621 07:51:19.076175 22082 leader_election.cc:288] T 00000000000000000000000000000000 P 751bf0e1893444738afe00d496b8b5c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:51:19.076275 22087 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 751bf0e1893444738afe00d496b8b5c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:19.076493 22087 raft_consensus.cc:686] T 00000000000000000000000000000000 P 751bf0e1893444738afe00d496b8b5c9 [term 1 LEADER]: Becoming Leader. State: Replica: 751bf0e1893444738afe00d496b8b5c9, State: Running, Role: LEADER
I20260621 07:51:19.076838 22087 consensus_queue.cc:237] T 00000000000000000000000000000000 P 751bf0e1893444738afe00d496b8b5c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751bf0e1893444738afe00d496b8b5c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41835 } }
I20260621 07:51:19.077309 22082 sys_catalog.cc:564] T 00000000000000000000000000000000 P 751bf0e1893444738afe00d496b8b5c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:51:19.077961 22088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 751bf0e1893444738afe00d496b8b5c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 751bf0e1893444738afe00d496b8b5c9. Latest consensus state: current_term: 1 leader_uuid: "751bf0e1893444738afe00d496b8b5c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751bf0e1893444738afe00d496b8b5c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41835 } } }
I20260621 07:51:19.078071 22088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 751bf0e1893444738afe00d496b8b5c9 [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:19.078364 22099 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:51:19.078316 22089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 751bf0e1893444738afe00d496b8b5c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "751bf0e1893444738afe00d496b8b5c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751bf0e1893444738afe00d496b8b5c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41835 } } }
I20260621 07:51:19.078521 22089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 751bf0e1893444738afe00d496b8b5c9 [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:19.081035 22099 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:51:19.085650 22099 catalog_manager.cc:1347] Generated new cluster ID: 91466463085847149180ae6b06fece75
I20260621 07:51:19.085740 22099 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:51:19.092199 22085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:19.079566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:51595 (local address 127.0.0.1:41835)
0621 07:51:19.079653 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:19.079656 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:19.084773 (+ 5117us) server_negotiation.cc:407] Connection header received
0621 07:51:19.084882 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:19.084884 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:19.084919 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:19.084995 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:19.085568 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:19.086210 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:19.087223 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:19.087513 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:19.087588 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:19.089039 (+ 1451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:19.089042 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:19.089045 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:19.089065 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:19.090460 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:19.091011 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:19.091014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:19.091015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:19.091087 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:19.091427 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:19.091429 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:19.091431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:19.091593 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:19.091720 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:19.091993 (+ 273us) server_negotiation.cc:299] Negotiation successful
0621 07:51:19.092071 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260621 07:51:19.098258 22099 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:51:19.099000 22099 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:51:19.107869 22099 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 751bf0e1893444738afe00d496b8b5c9: Generated new TSK 0
I20260621 07:51:19.108436 22099 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:51:19.116346 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 22014
W20260621 07:51:19.120448 21999 connection.cc:535] client connection to 127.0.0.1:41835 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41835): Connection reset by peer (error 104)
2026-06-21T07:51:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1246 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028028522580-771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:51:19 dist-test-slave-lzx3 krb5kdc[22113](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:51:19 dist-test-slave-lzx3 krb5kdc[22113](info): set up 2 sockets
Jun 21 07:51:19 dist-test-slave-lzx3 krb5kdc[22113](info): commencing operation
krb5kdc: starting...
W20260621 07:51:20.060518 771 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028028522580-771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028028522580-771-0/krb5kdc/test-user.keytab.
Jun 21 07:51:20 dist-test-slave-lzx3 krb5kdc[22113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:51:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:51:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 07:51:20.163288 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36363
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:20.256974 22126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:20.257165 22126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:20.257200 22126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:20.260946 22126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:51:20.260991 22126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:20.261010 22126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:20.261026 22126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:51:20.261062 22126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:51:20.265867 22126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36363
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:20.266764 22126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:20.267665 22126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:20.275197 22134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:20.275251 22131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:20.275266 22132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:20.275344 22126 server_base.cc:1034] running on GCE node
I20260621 07:51:20.275993 22126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:20.276963 22126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:20.278177 22126 hybrid_clock.cc:648] HybridClock initialized: now 1782028280278146 us; error 41 us; skew 500 ppm
Jun 21 07:51:20 dist-test-slave-lzx3 krb5kdc[22113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028280, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:20.285084 22126 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 07:51:20.285739 22126 webserver.cc:460] Webserver started at http://127.0.0.1:45229/ using document root <none> and password file <none>
I20260621 07:51:20.286145 22126 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:20.286193 22126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:20.286324 22126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:20.287976 22126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "163005d0f60541108d00dd1d21a52eb1"
format_stamp: "Formatted at 2026-06-21 07:51:20 on dist-test-slave-lzx3"
server_key: "8a439589cdb2f74ee45883ee04998103"
server_key_iv: "0f4cf670d4e228ae1cb08574193d3aef"
server_key_version: "encryptionkey@0"
I20260621 07:51:20.288408 22126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "163005d0f60541108d00dd1d21a52eb1"
format_stamp: "Formatted at 2026-06-21 07:51:20 on dist-test-slave-lzx3"
server_key: "8a439589cdb2f74ee45883ee04998103"
server_key_iv: "0f4cf670d4e228ae1cb08574193d3aef"
server_key_version: "encryptionkey@0"
I20260621 07:51:20.291464 22126 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260621 07:51:20.293463 22141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:20.294461 22126 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:51:20.294545 22126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "163005d0f60541108d00dd1d21a52eb1"
format_stamp: "Formatted at 2026-06-21 07:51:20 on dist-test-slave-lzx3"
server_key: "8a439589cdb2f74ee45883ee04998103"
server_key_iv: "0f4cf670d4e228ae1cb08574193d3aef"
server_key_version: "encryptionkey@0"
I20260621 07:51:20.294612 22126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:20.312052 22126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:20.314283 22126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:20.314407 22126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:20.319873 22126 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36363
I20260621 07:51:20.319944 22193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36363 every 8 connection(s)
I20260621 07:51:20.320688 22126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:51:20.323035 22194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:20.327574 22194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 163005d0f60541108d00dd1d21a52eb1: Bootstrap starting.
I20260621 07:51:20.328068 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 22126
I20260621 07:51:20.328234 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:51:20.328533 771 external_mini_cluster.cc:1428] Setting key a069bfa3e798dd64ce72a9c42eb3ab29
I20260621 07:51:20.330078 22194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 163005d0f60541108d00dd1d21a52eb1: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:20.330756 22194 log.cc:824] T 00000000000000000000000000000000 P 163005d0f60541108d00dd1d21a52eb1: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:20.332653 22194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 163005d0f60541108d00dd1d21a52eb1: No bootstrap required, opened a new log
I20260621 07:51:20.336328 22194 raft_consensus.cc:348] T 00000000000000000000000000000000 P 163005d0f60541108d00dd1d21a52eb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "163005d0f60541108d00dd1d21a52eb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36363 } }
I20260621 07:51:20.336544 22194 raft_consensus.cc:374] T 00000000000000000000000000000000 P 163005d0f60541108d00dd1d21a52eb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:20.336642 22194 raft_consensus.cc:729] T 00000000000000000000000000000000 P 163005d0f60541108d00dd1d21a52eb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 163005d0f60541108d00dd1d21a52eb1, State: Initialized, Role: FOLLOWER
I20260621 07:51:20.336928 22197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:20.329638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:36975 (local address 127.0.0.1:36363)
0621 07:51:20.330023 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:20.330037 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:20.330070 (+ 33us) server_negotiation.cc:407] Connection header received
0621 07:51:20.330722 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:20.330738 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:20.331152 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:20.331536 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:20.331846 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:20.332822 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:20.333902 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:20.334324 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:20.334431 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:20.335190 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:20.335194 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:20.335204 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:51:20.335539 (+ 335us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:20.335588 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:51:20.335608 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:20.335703 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:20.335930 (+ 227us) server_negotiation.cc:299] Negotiation successful
0621 07:51:20.336077 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":140,"threads_started":1}
I20260621 07:51:20.337101 22194 consensus_queue.cc:260] T 00000000000000000000000000000000 P 163005d0f60541108d00dd1d21a52eb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "163005d0f60541108d00dd1d21a52eb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36363 } }
I20260621 07:51:20.337234 22194 raft_consensus.cc:388] T 00000000000000000000000000000000 P 163005d0f60541108d00dd1d21a52eb1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:51:20.337291 22194 raft_consensus.cc:482] T 00000000000000000000000000000000 P 163005d0f60541108d00dd1d21a52eb1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:51:20.337378 22194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 163005d0f60541108d00dd1d21a52eb1 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:20.338179 22194 raft_consensus.cc:504] T 00000000000000000000000000000000 P 163005d0f60541108d00dd1d21a52eb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "163005d0f60541108d00dd1d21a52eb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36363 } }
I20260621 07:51:20.338519 22194 leader_election.cc:302] T 00000000000000000000000000000000 P 163005d0f60541108d00dd1d21a52eb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 163005d0f60541108d00dd1d21a52eb1; no voters:
I20260621 07:51:20.338814 22194 leader_election.cc:288] T 00000000000000000000000000000000 P 163005d0f60541108d00dd1d21a52eb1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:51:20.338917 22199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 163005d0f60541108d00dd1d21a52eb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:20.339143 22199 raft_consensus.cc:686] T 00000000000000000000000000000000 P 163005d0f60541108d00dd1d21a52eb1 [term 1 LEADER]: Becoming Leader. State: Replica: 163005d0f60541108d00dd1d21a52eb1, State: Running, Role: LEADER
I20260621 07:51:20.339557 22199 consensus_queue.cc:237] T 00000000000000000000000000000000 P 163005d0f60541108d00dd1d21a52eb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "163005d0f60541108d00dd1d21a52eb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36363 } }
I20260621 07:51:20.339720 771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:51:20.339984 22194 sys_catalog.cc:564] T 00000000000000000000000000000000 P 163005d0f60541108d00dd1d21a52eb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:51:20.341084 22201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 163005d0f60541108d00dd1d21a52eb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 163005d0f60541108d00dd1d21a52eb1. Latest consensus state: current_term: 1 leader_uuid: "163005d0f60541108d00dd1d21a52eb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "163005d0f60541108d00dd1d21a52eb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36363 } } }
I20260621 07:51:20.341233 22201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 163005d0f60541108d00dd1d21a52eb1 [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:20.341727 22212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:51:20.341723 22200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 163005d0f60541108d00dd1d21a52eb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "163005d0f60541108d00dd1d21a52eb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "163005d0f60541108d00dd1d21a52eb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36363 } } }
I20260621 07:51:20.341923 22200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 163005d0f60541108d00dd1d21a52eb1 [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:20.344655 22212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:51:20.347159 22197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:20.342671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:51203 (local address 127.0.0.1:36363)
0621 07:51:20.342777 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:20.342781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:20.343072 (+ 291us) server_negotiation.cc:407] Connection header received
0621 07:51:20.343161 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:20.343164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:20.343203 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:20.343280 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:20.343629 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:20.344279 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:20.345359 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:20.345647 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:20.345717 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:20.346660 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:20.346663 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:20.346665 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:51:20.346677 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:20.346699 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:51:20.346715 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:20.346818 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:20.346986 (+ 168us) server_negotiation.cc:299] Negotiation successful
0621 07:51:20.347016 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260621 07:51:20.349663 22212 catalog_manager.cc:1347] Generated new cluster ID: 1fd0a6972b274c078d81d6f95637c866
I20260621 07:51:20.349722 22212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:51:20.358158 22212 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:51:20.359375 22212 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:51:20.365916 22212 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 163005d0f60541108d00dd1d21a52eb1: Generated new TSK 0
I20260621 07:51:20.366417 22212 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:51:20.372807 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 22126
W20260621 07:51:20.377266 22111 connection.cc:535] client connection to 127.0.0.1:36363 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36363): Connection reset by peer (error 104)
2026-06-21T07:51:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1256 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028028522580-771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:51:20 dist-test-slave-lzx3 krb5kdc[22225](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:51:20 dist-test-slave-lzx3 krb5kdc[22225](info): set up 2 sockets
Jun 21 07:51:20 dist-test-slave-lzx3 krb5kdc[22225](info): commencing operation
krb5kdc: starting...
W20260621 07:51:21.328739 771 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028028522580-771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028028522580-771-0/krb5kdc/test-user.keytab.
Jun 21 07:51:21 dist-test-slave-lzx3 krb5kdc[22225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:51:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:51:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 07:51:21.442852 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45427
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:21.546802 22238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:21.547025 22238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:21.547127 22238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:21.551030 22238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:51:21.551087 22238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:21.551112 22238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:21.551160 22238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:51:21.551194 22238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:51:21.556135 22238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45427
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028028522580-771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:21.557070 22238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:21.558017 22238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:21.566159 22244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:21.566159 22246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:21.566184 22243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:21.566406 22238 server_base.cc:1034] running on GCE node
I20260621 07:51:21.567026 22238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:21.568020 22238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:21.569200 22238 hybrid_clock.cc:648] HybridClock initialized: now 1782028281569166 us; error 48 us; skew 500 ppm
Jun 21 07:51:21 dist-test-slave-lzx3 krb5kdc[22225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:21.571961 22238 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 07:51:21.572873 22238 webserver.cc:460] Webserver started at http://127.0.0.1:36819/ using document root <none> and password file <none>
I20260621 07:51:21.573377 22238 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:21.573432 22238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:21.573645 22238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:21.575516 22238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "90180e1b658045e6b1d958ffdcd6a194"
format_stamp: "Formatted at 2026-06-21 07:51:21 on dist-test-slave-lzx3"
server_key: "ce0b6afef3153447f2da49c95c7015e1"
server_key_iv: "20d19b9ed24a4a6cb0a92e70a511101e"
server_key_version: "encryptionkey@0"
I20260621 07:51:21.576084 22238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "90180e1b658045e6b1d958ffdcd6a194"
format_stamp: "Formatted at 2026-06-21 07:51:21 on dist-test-slave-lzx3"
server_key: "ce0b6afef3153447f2da49c95c7015e1"
server_key_iv: "20d19b9ed24a4a6cb0a92e70a511101e"
server_key_version: "encryptionkey@0"
I20260621 07:51:21.579491 22238 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260621 07:51:21.581797 22253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:21.582922 22238 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 07:51:21.583058 22238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "90180e1b658045e6b1d958ffdcd6a194"
format_stamp: "Formatted at 2026-06-21 07:51:21 on dist-test-slave-lzx3"
server_key: "ce0b6afef3153447f2da49c95c7015e1"
server_key_iv: "20d19b9ed24a4a6cb0a92e70a511101e"
server_key_version: "encryptionkey@0"
I20260621 07:51:21.583163 22238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:21.587420 22238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:21.590023 22238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:21.590201 22238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:21.596539 22238 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45427
I20260621 07:51:21.596645 22305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45427 every 8 connection(s)
I20260621 07:51:21.597429 22238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:51:21.597894 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 22238
I20260621 07:51:21.598050 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:51:21.598294 771 external_mini_cluster.cc:1428] Setting key e42140d4d93f1e6dd8f063e3765a3fcb
I20260621 07:51:21.600446 22306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:21.603360 22309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:21.599527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:42095 (local address 127.0.0.1:45427)
0621 07:51:21.599942 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:21.599960 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:21.599998 (+ 38us) server_negotiation.cc:407] Connection header received
0621 07:51:21.600807 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:21.600840 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:21.601158 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:21.601489 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:21.602127 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:21.602133 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:21.602148 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:51:21.602163 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:21.602227 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:51:21.602251 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:21.602379 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:21.602443 (+ 64us) server_negotiation.cc:299] Negotiation successful
0621 07:51:21.602575 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":94,"threads_started":1}
I20260621 07:51:21.605958 771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:51:21.606518 22306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90180e1b658045e6b1d958ffdcd6a194: Bootstrap starting.
I20260621 07:51:21.608835 22306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90180e1b658045e6b1d958ffdcd6a194: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:21.609558 22309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:21.607957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:51075 (local address 127.0.0.1:45427)
0621 07:51:21.608078 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:21.608083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:21.608273 (+ 190us) server_negotiation.cc:407] Connection header received
0621 07:51:21.608383 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:21.608385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:21.608424 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:21.608484 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:21.609172 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:21.609175 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:21.609178 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:51:21.609189 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:21.609214 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:51:21.609232 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:21.609329 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:21.609401 (+ 72us) server_negotiation.cc:299] Negotiation successful
0621 07:51:21.609440 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 07:51:21.609572 22306 log.cc:824] T 00000000000000000000000000000000 P 90180e1b658045e6b1d958ffdcd6a194: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:21.611428 22306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90180e1b658045e6b1d958ffdcd6a194: No bootstrap required, opened a new log
I20260621 07:51:21.614724 22306 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90180e1b658045e6b1d958ffdcd6a194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90180e1b658045e6b1d958ffdcd6a194" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45427 } }
I20260621 07:51:21.614908 22306 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90180e1b658045e6b1d958ffdcd6a194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:21.614985 22306 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90180e1b658045e6b1d958ffdcd6a194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90180e1b658045e6b1d958ffdcd6a194, State: Initialized, Role: FOLLOWER
I20260621 07:51:21.615394 22306 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90180e1b658045e6b1d958ffdcd6a194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90180e1b658045e6b1d958ffdcd6a194" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45427 } }
I20260621 07:51:21.615525 22306 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90180e1b658045e6b1d958ffdcd6a194 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:51:21.615609 22306 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90180e1b658045e6b1d958ffdcd6a194 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:51:21.615722 22306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90180e1b658045e6b1d958ffdcd6a194 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:21.616739 22306 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90180e1b658045e6b1d958ffdcd6a194 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90180e1b658045e6b1d958ffdcd6a194" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45427 } }
I20260621 07:51:21.617067 22306 leader_election.cc:302] T 00000000000000000000000000000000 P 90180e1b658045e6b1d958ffdcd6a194 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90180e1b658045e6b1d958ffdcd6a194; no voters:
I20260621 07:51:21.617365 22306 leader_election.cc:288] T 00000000000000000000000000000000 P 90180e1b658045e6b1d958ffdcd6a194 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:51:21.617520 22316 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90180e1b658045e6b1d958ffdcd6a194 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:21.617779 22316 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90180e1b658045e6b1d958ffdcd6a194 [term 1 LEADER]: Becoming Leader. State: Replica: 90180e1b658045e6b1d958ffdcd6a194, State: Running, Role: LEADER
I20260621 07:51:21.618088 22316 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90180e1b658045e6b1d958ffdcd6a194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90180e1b658045e6b1d958ffdcd6a194" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45427 } }
I20260621 07:51:21.618491 22306 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90180e1b658045e6b1d958ffdcd6a194 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:51:21.619664 22318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90180e1b658045e6b1d958ffdcd6a194 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90180e1b658045e6b1d958ffdcd6a194. Latest consensus state: current_term: 1 leader_uuid: "90180e1b658045e6b1d958ffdcd6a194" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90180e1b658045e6b1d958ffdcd6a194" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45427 } } }
I20260621 07:51:21.619789 22318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90180e1b658045e6b1d958ffdcd6a194 [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:21.619969 22317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90180e1b658045e6b1d958ffdcd6a194 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90180e1b658045e6b1d958ffdcd6a194" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90180e1b658045e6b1d958ffdcd6a194" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45427 } } }
I20260621 07:51:21.620127 22317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90180e1b658045e6b1d958ffdcd6a194 [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:21.620419 22327 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:51:21.622931 22327 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:51:21.628232 22327 catalog_manager.cc:1347] Generated new cluster ID: 986c0900cdf641d09b1dafeaff4900bb
I20260621 07:51:21.628329 22327 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:51:21.635906 22327 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:51:21.636178 22327 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:51:21.646522 22327 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90180e1b658045e6b1d958ffdcd6a194: Generated new TSK 0
I20260621 07:51:21.647109 22327 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:51:21.653167 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 22238
2026-06-21T07:51:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1280 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028028522580-771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:51:21 dist-test-slave-lzx3 krb5kdc[22337](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:51:21 dist-test-slave-lzx3 krb5kdc[22337](info): set up 2 sockets
Jun 21 07:51:21 dist-test-slave-lzx3 krb5kdc[22337](info): commencing operation
krb5kdc: starting...
W20260621 07:51:22.684715 771 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028028522580-771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028028522580-771-0/krb5kdc/test-user.keytab.
Jun 21 07:51:22 dist-test-slave-lzx3 krb5kdc[22337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:51:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:51:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.192.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
I20260621 07:51:22.795969 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:33051
--webserver_interface=127.0.192.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:35295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:33051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:22.896996 22350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:22.897251 22350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:22.897320 22350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:22.901224 22350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:51:22.901310 22350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:22.901342 22350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:22.901381 22350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:51:22.901408 22350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:51:22.906316 22350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:35295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:33051
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:33051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:22.907210 22350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:22.908102 22350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:22.915798 22355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:22.915879 22350 server_base.cc:1034] running on GCE node
W20260621 07:51:22.915810 22358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:22.915803 22356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:22.916436 22350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:22.917460 22350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:22.918645 22350 hybrid_clock.cc:648] HybridClock initialized: now 1782028282918629 us; error 27 us; skew 500 ppm
Jun 21 07:51:22 dist-test-slave-lzx3 krb5kdc[22337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:22.921139 22350 init.cc:377] Logged in from keytab as kudu/127.0.192.254@KRBTEST.COM (short username kudu)
I20260621 07:51:22.921933 22350 webserver.cc:460] Webserver started at http://127.0.192.254:35835/ using document root <none> and password file <none>
I20260621 07:51:22.922411 22350 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:22.922511 22350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:22.922722 22350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:22.924496 22350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "b7dae62811f141e0ac75feb6aec82f96"
format_stamp: "Formatted at 2026-06-21 07:51:22 on dist-test-slave-lzx3"
server_key: "aac296d0c0642db309320f453c979c93"
server_key_iv: "5fc343cbfb04f16a3141e296d5bb211d"
server_key_version: "encryptionkey@0"
I20260621 07:51:22.925127 22350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "b7dae62811f141e0ac75feb6aec82f96"
format_stamp: "Formatted at 2026-06-21 07:51:22 on dist-test-slave-lzx3"
server_key: "aac296d0c0642db309320f453c979c93"
server_key_iv: "5fc343cbfb04f16a3141e296d5bb211d"
server_key_version: "encryptionkey@0"
I20260621 07:51:22.928493 22350 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:51:22.930728 22365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:22.931814 22350 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260621 07:51:22.931949 22350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "b7dae62811f141e0ac75feb6aec82f96"
format_stamp: "Formatted at 2026-06-21 07:51:22 on dist-test-slave-lzx3"
server_key: "aac296d0c0642db309320f453c979c93"
server_key_iv: "5fc343cbfb04f16a3141e296d5bb211d"
server_key_version: "encryptionkey@0"
I20260621 07:51:22.932046 22350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:22.948343 22350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:22.950744 22350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:22.950918 22350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:22.956709 22350 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:33051
I20260621 07:51:22.956719 22417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:33051 every 8 connection(s)
I20260621 07:51:22.957533 22350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:51:22.959836 22418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:22.961059 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 22350
I20260621 07:51:22.961197 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:51:22.961438 771 external_mini_cluster.cc:1428] Setting key 80e8bcfaea4e07992318256f16bdb6b9
I20260621 07:51:22.965476 22418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7dae62811f141e0ac75feb6aec82f96: Bootstrap starting.
I20260621 07:51:22.967669 22418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7dae62811f141e0ac75feb6aec82f96: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:22.968393 22418 log.cc:824] T 00000000000000000000000000000000 P b7dae62811f141e0ac75feb6aec82f96: Log is configured to *not* fsync() on all Append() calls
Jun 21 07:51:22 dist-test-slave-lzx3 krb5kdc[22337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:51:22.970326 22418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7dae62811f141e0ac75feb6aec82f96: No bootstrap required, opened a new log
I20260621 07:51:22.974566 22418 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7dae62811f141e0ac75feb6aec82f96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7dae62811f141e0ac75feb6aec82f96" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 33051 } }
I20260621 07:51:22.974817 22418 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7dae62811f141e0ac75feb6aec82f96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:22.974862 22421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:22.962604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55073 (local address 127.0.192.254:33051)
0621 07:51:22.962933 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:22.962948 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:22.962982 (+ 34us) server_negotiation.cc:407] Connection header received
0621 07:51:22.963682 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:22.963701 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:22.964046 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:22.964349 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:22.964976 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:22.966176 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:22.967018 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:22.967595 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:22.967737 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:22.970386 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:22.970392 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:22.970407 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:22.970434 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:22.972167 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:22.972678 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:22.972683 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:22.972688 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:22.972807 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:22.973091 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:22.973094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:22.973096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:22.973505 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:22.973706 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:22.974013 (+ 307us) server_negotiation.cc:299] Negotiation successful
0621 07:51:22.974192 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":83,"threads_started":1}
I20260621 07:51:22.974871 22418 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7dae62811f141e0ac75feb6aec82f96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7dae62811f141e0ac75feb6aec82f96, State: Initialized, Role: FOLLOWER
I20260621 07:51:22.975314 22418 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7dae62811f141e0ac75feb6aec82f96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7dae62811f141e0ac75feb6aec82f96" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 33051 } }
I20260621 07:51:22.975423 22418 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7dae62811f141e0ac75feb6aec82f96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:51:22.975498 22418 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7dae62811f141e0ac75feb6aec82f96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:51:22.975591 22418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7dae62811f141e0ac75feb6aec82f96 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:22.976516 22418 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7dae62811f141e0ac75feb6aec82f96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7dae62811f141e0ac75feb6aec82f96" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 33051 } }
I20260621 07:51:22.976814 22418 leader_election.cc:302] T 00000000000000000000000000000000 P b7dae62811f141e0ac75feb6aec82f96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7dae62811f141e0ac75feb6aec82f96; no voters:
I20260621 07:51:22.977149 22418 leader_election.cc:288] T 00000000000000000000000000000000 P b7dae62811f141e0ac75feb6aec82f96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:51:22.977245 22423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7dae62811f141e0ac75feb6aec82f96 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:22.977732 22423 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7dae62811f141e0ac75feb6aec82f96 [term 1 LEADER]: Becoming Leader. State: Replica: b7dae62811f141e0ac75feb6aec82f96, State: Running, Role: LEADER
I20260621 07:51:22.977829 771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:51:22.978044 22423 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7dae62811f141e0ac75feb6aec82f96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7dae62811f141e0ac75feb6aec82f96" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 33051 } }
I20260621 07:51:22.978331 22418 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7dae62811f141e0ac75feb6aec82f96 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:51:22.979099 22423 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7dae62811f141e0ac75feb6aec82f96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7dae62811f141e0ac75feb6aec82f96. Latest consensus state: current_term: 1 leader_uuid: "b7dae62811f141e0ac75feb6aec82f96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7dae62811f141e0ac75feb6aec82f96" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 33051 } } }
I20260621 07:51:22.979199 22423 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7dae62811f141e0ac75feb6aec82f96 [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:22.979458 22424 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7dae62811f141e0ac75feb6aec82f96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7dae62811f141e0ac75feb6aec82f96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7dae62811f141e0ac75feb6aec82f96" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 33051 } } }
I20260621 07:51:22.979616 22424 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7dae62811f141e0ac75feb6aec82f96 [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:22.979555 22436 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:51:22.982501 22436 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:51:22.987703 22421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:22.979913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33723 (local address 127.0.192.254:33051)
0621 07:51:22.979998 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:22.980002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:22.980825 (+ 823us) server_negotiation.cc:407] Connection header received
0621 07:51:22.980939 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:22.980941 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:22.980975 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:22.981043 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:22.981695 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:22.982319 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:22.983254 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:22.983515 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:22.983610 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:22.984896 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:22.984899 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:22.984901 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:22.984921 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:22.986214 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:22.986635 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:22.986638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:22.986640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:22.986711 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:22.986996 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:22.986998 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:22.986999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:22.987136 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:22.987261 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:22.987486 (+ 225us) server_negotiation.cc:299] Negotiation successful
0621 07:51:22.987560 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260621 07:51:22.987809 22436 catalog_manager.cc:1347] Generated new cluster ID: fa2a48d94d2d44449b2bfd8271dd0dac
I20260621 07:51:22.987866 22436 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:51:23.001078 22436 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:51:23.001775 22436 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:51:23.005656 22436 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7dae62811f141e0ac75feb6aec82f96: Generated new TSK 0
I20260621 07:51:23.006137 22436 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:51:23.013648 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 22350
W20260621 07:51:23.017995 22335 connection.cc:535] client connection to 127.0.192.254:33051 recv error: Network error: failed to read from TLS socket (remote: 127.0.192.254:33051): Connection reset by peer (error 104)
2026-06-21T07:51:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1359 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028028522580-771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:51:23 dist-test-slave-lzx3 krb5kdc[22449](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:51:23 dist-test-slave-lzx3 krb5kdc[22449](info): set up 2 sockets
Jun 21 07:51:23 dist-test-slave-lzx3 krb5kdc[22449](info): commencing operation
krb5kdc: starting...
W20260621 07:51:23.972961 771 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028028522580-771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028028522580-771-0/krb5kdc/test-user.keytab.
Jun 21 07:51:24 dist-test-slave-lzx3 krb5kdc[22449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:51:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:51:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.192.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
I20260621 07:51:24.079313 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:35141
--webserver_interface=127.0.192.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:43627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:35141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:24.177909 22462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:24.178128 22462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:24.178191 22462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:24.181758 22462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:51:24.181823 22462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:24.181855 22462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:24.181901 22462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:51:24.181931 22462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:51:24.186257 22462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:43627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:35141
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:35141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:24.187027 22462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:24.187819 22462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:24.195080 22468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:24.195138 22462 server_base.cc:1034] running on GCE node
W20260621 07:51:24.195080 22470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:24.195096 22467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:24.195676 22462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:24.196666 22462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:24.197824 22462 hybrid_clock.cc:648] HybridClock initialized: now 1782028284197807 us; error 30 us; skew 500 ppm
Jun 21 07:51:24 dist-test-slave-lzx3 krb5kdc[22449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:24.200371 22462 init.cc:377] Logged in from keytab as kudu/127.0.192.254@KRBTEST.COM (short username kudu)
I20260621 07:51:24.201195 22462 webserver.cc:460] Webserver started at http://127.0.192.254:42629/ using document root <none> and password file <none>
I20260621 07:51:24.201699 22462 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:24.201807 22462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:24.202010 22462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:24.203786 22462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "3a303072411449c6ba87bd6c3d6f16ef"
format_stamp: "Formatted at 2026-06-21 07:51:24 on dist-test-slave-lzx3"
server_key: "f2dbc74cdb41937c723c61336b17340b"
server_key_iv: "a0b1a0289b6480ac2ed11472fed90165"
server_key_version: "encryptionkey@0"
I20260621 07:51:24.204296 22462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "3a303072411449c6ba87bd6c3d6f16ef"
format_stamp: "Formatted at 2026-06-21 07:51:24 on dist-test-slave-lzx3"
server_key: "f2dbc74cdb41937c723c61336b17340b"
server_key_iv: "a0b1a0289b6480ac2ed11472fed90165"
server_key_version: "encryptionkey@0"
I20260621 07:51:24.207597 22462 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260621 07:51:24.209875 22477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:24.210983 22462 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 07:51:24.211112 22462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "3a303072411449c6ba87bd6c3d6f16ef"
format_stamp: "Formatted at 2026-06-21 07:51:24 on dist-test-slave-lzx3"
server_key: "f2dbc74cdb41937c723c61336b17340b"
server_key_iv: "a0b1a0289b6480ac2ed11472fed90165"
server_key_version: "encryptionkey@0"
I20260621 07:51:24.211215 22462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:24.223112 22462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:24.225796 22462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:24.225982 22462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:24.232523 22462 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:35141
I20260621 07:51:24.232633 22529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:35141 every 8 connection(s)
I20260621 07:51:24.233417 22462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:51:24.234611 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 22462
I20260621 07:51:24.234746 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:51:24.234998 771 external_mini_cluster.cc:1428] Setting key d8f1ed66f16bb95658164b19413d1e21
I20260621 07:51:24.236150 22530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:24.242333 22530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a303072411449c6ba87bd6c3d6f16ef: Bootstrap starting.
I20260621 07:51:24.244153 22533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:24.236266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56195 (local address 127.0.192.254:35141)
0621 07:51:24.236690 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:24.236708 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:24.236748 (+ 40us) server_negotiation.cc:407] Connection header received
0621 07:51:24.237549 (+ 801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:24.237570 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:24.237964 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:24.238295 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:24.238662 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:24.240111 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:24.240992 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:24.241632 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:24.241768 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:24.242492 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:24.242496 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:24.242510 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:51:24.242924 (+ 414us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:24.242979 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:51:24.243005 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:24.243118 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:24.243290 (+ 172us) server_negotiation.cc:299] Negotiation successful
0621 07:51:24.243418 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":131,"threads_started":1}
I20260621 07:51:24.244638 22530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a303072411449c6ba87bd6c3d6f16ef: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:24.245378 22530 log.cc:824] T 00000000000000000000000000000000 P 3a303072411449c6ba87bd6c3d6f16ef: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:24.246786 771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:51:24.247190 22530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a303072411449c6ba87bd6c3d6f16ef: No bootstrap required, opened a new log
I20260621 07:51:24.251015 22530 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a303072411449c6ba87bd6c3d6f16ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a303072411449c6ba87bd6c3d6f16ef" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 35141 } }
I20260621 07:51:24.251241 22530 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a303072411449c6ba87bd6c3d6f16ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:24.251346 22530 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a303072411449c6ba87bd6c3d6f16ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a303072411449c6ba87bd6c3d6f16ef, State: Initialized, Role: FOLLOWER
I20260621 07:51:24.251757 22530 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a303072411449c6ba87bd6c3d6f16ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a303072411449c6ba87bd6c3d6f16ef" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 35141 } }
I20260621 07:51:24.251906 22530 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a303072411449c6ba87bd6c3d6f16ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:51:24.251989 22530 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a303072411449c6ba87bd6c3d6f16ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:51:24.252110 22530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a303072411449c6ba87bd6c3d6f16ef [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:24.252444 22533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:24.248660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39309 (local address 127.0.192.254:35141)
0621 07:51:24.248753 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:24.248756 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:24.248908 (+ 152us) server_negotiation.cc:407] Connection header received
0621 07:51:24.249021 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:24.249023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:24.249068 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:24.249146 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:24.249440 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:24.249884 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:24.250941 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:24.251125 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:24.251190 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:24.251990 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:24.251993 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:24.251998 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:51:24.252009 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:24.252046 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:51:24.252058 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:24.252154 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:24.252281 (+ 127us) server_negotiation.cc:299] Negotiation successful
0621 07:51:24.252306 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260621 07:51:24.253031 22530 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a303072411449c6ba87bd6c3d6f16ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a303072411449c6ba87bd6c3d6f16ef" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 35141 } }
I20260621 07:51:24.253345 22530 leader_election.cc:302] T 00000000000000000000000000000000 P 3a303072411449c6ba87bd6c3d6f16ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a303072411449c6ba87bd6c3d6f16ef; no voters:
I20260621 07:51:24.253634 22530 leader_election.cc:288] T 00000000000000000000000000000000 P 3a303072411449c6ba87bd6c3d6f16ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:51:24.253696 22540 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a303072411449c6ba87bd6c3d6f16ef [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:24.253921 22540 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a303072411449c6ba87bd6c3d6f16ef [term 1 LEADER]: Becoming Leader. State: Replica: 3a303072411449c6ba87bd6c3d6f16ef, State: Running, Role: LEADER
I20260621 07:51:24.254220 22540 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a303072411449c6ba87bd6c3d6f16ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a303072411449c6ba87bd6c3d6f16ef" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 35141 } }
I20260621 07:51:24.254622 22530 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a303072411449c6ba87bd6c3d6f16ef [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:51:24.255816 22542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a303072411449c6ba87bd6c3d6f16ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a303072411449c6ba87bd6c3d6f16ef. Latest consensus state: current_term: 1 leader_uuid: "3a303072411449c6ba87bd6c3d6f16ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a303072411449c6ba87bd6c3d6f16ef" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 35141 } } }
I20260621 07:51:24.255823 22541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a303072411449c6ba87bd6c3d6f16ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a303072411449c6ba87bd6c3d6f16ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a303072411449c6ba87bd6c3d6f16ef" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 35141 } } }
I20260621 07:51:24.255983 22541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a303072411449c6ba87bd6c3d6f16ef [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:24.256237 22542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a303072411449c6ba87bd6c3d6f16ef [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:24.256301 22550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:51:24.259331 22550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:51:24.264106 22550 catalog_manager.cc:1347] Generated new cluster ID: dcf62562ae674f5db031f5cca7456911
I20260621 07:51:24.264159 22550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:51:24.301062 22550 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:51:24.302064 22550 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:51:24.306334 22550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a303072411449c6ba87bd6c3d6f16ef: Generated new TSK 0
I20260621 07:51:24.306900 22550 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:51:24.312901 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 22462
W20260621 07:51:24.318009 22447 connection.cc:535] client connection to 127.0.192.254:35141 recv error: Network error: failed to read from TLS socket (remote: 127.0.192.254:35141): Connection reset by peer (error 104)
2026-06-21T07:51:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1299 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028028522580-771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:51:24 dist-test-slave-lzx3 krb5kdc[22561](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:51:24 dist-test-slave-lzx3 krb5kdc[22561](info): set up 2 sockets
Jun 21 07:51:24 dist-test-slave-lzx3 krb5kdc[22561](info): commencing operation
krb5kdc: starting...
W20260621 07:51:25.256349 771 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028028522580-771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028028522580-771-0/krb5kdc/test-user.keytab.
Jun 21 07:51:25 dist-test-slave-lzx3 krb5kdc[22561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:51:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:51:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.192.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
I20260621 07:51:25.358745 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:37763
--webserver_interface=127.0.192.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:44707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:37763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:25.456512 22574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:25.456751 22574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:25.456825 22574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:25.460420 22574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:51:25.460464 22574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:25.460479 22574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:25.460510 22574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:51:25.460525 22574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:51:25.464943 22574 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:44707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:37763
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:37763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:25.465735 22574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:25.466594 22574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 07:51:25.473798 22574 server_base.cc:1034] running on GCE node
W20260621 07:51:25.473757 22579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:25.473773 22582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:25.473757 22580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:25.474514 22574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:25.475363 22574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:25.476536 22574 hybrid_clock.cc:648] HybridClock initialized: now 1782028285476516 us; error 35 us; skew 500 ppm
Jun 21 07:51:25 dist-test-slave-lzx3 krb5kdc[22561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028285, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:25.481995 22574 init.cc:377] Logged in from keytab as kudu/127.0.192.254@KRBTEST.COM (short username kudu)
I20260621 07:51:25.482712 22574 webserver.cc:460] Webserver started at http://127.0.192.254:40575/ using document root <none> and password file <none>
I20260621 07:51:25.483206 22574 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:25.483296 22574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:25.483493 22574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:25.485157 22574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "c2d109f8bd2c4300b64e736c837f3871"
format_stamp: "Formatted at 2026-06-21 07:51:25 on dist-test-slave-lzx3"
server_key: "c5ced198c09db985483b0812a5c5ed4f"
server_key_iv: "5f0294e441044e286ceada17e9b5d7f9"
server_key_version: "encryptionkey@0"
I20260621 07:51:25.485631 22574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "c2d109f8bd2c4300b64e736c837f3871"
format_stamp: "Formatted at 2026-06-21 07:51:25 on dist-test-slave-lzx3"
server_key: "c5ced198c09db985483b0812a5c5ed4f"
server_key_iv: "5f0294e441044e286ceada17e9b5d7f9"
server_key_version: "encryptionkey@0"
I20260621 07:51:25.488524 22574 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260621 07:51:25.490911 22589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:25.491895 22574 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:51:25.492028 22574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "c2d109f8bd2c4300b64e736c837f3871"
format_stamp: "Formatted at 2026-06-21 07:51:25 on dist-test-slave-lzx3"
server_key: "c5ced198c09db985483b0812a5c5ed4f"
server_key_iv: "5f0294e441044e286ceada17e9b5d7f9"
server_key_version: "encryptionkey@0"
I20260621 07:51:25.492123 22574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:25.495788 22574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:25.498485 22574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:25.498653 22574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:25.504508 22574 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:37763
I20260621 07:51:25.504585 22641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:37763 every 8 connection(s)
I20260621 07:51:25.505276 22574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:51:25.507455 22642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:25.512010 22642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2d109f8bd2c4300b64e736c837f3871: Bootstrap starting.
I20260621 07:51:25.513417 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 22574
I20260621 07:51:25.513537 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:51:25.513810 771 external_mini_cluster.cc:1428] Setting key efe4fbb2eab793af621122388fefc765
I20260621 07:51:25.514187 22642 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2d109f8bd2c4300b64e736c837f3871: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:25.514936 22642 log.cc:824] T 00000000000000000000000000000000 P c2d109f8bd2c4300b64e736c837f3871: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:25.516601 22642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2d109f8bd2c4300b64e736c837f3871: No bootstrap required, opened a new log
I20260621 07:51:25.518165 22645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:25.514713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53237 (local address 127.0.192.254:37763)
0621 07:51:25.515112 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:25.515123 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:25.515152 (+ 29us) server_negotiation.cc:407] Connection header received
0621 07:51:25.515813 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:25.515830 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:25.516095 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:25.516423 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:25.517134 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:25.517138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:25.517148 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:51:25.517169 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:25.517227 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:51:25.517242 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:25.517370 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:25.517469 (+ 99us) server_negotiation.cc:299] Negotiation successful
0621 07:51:25.517615 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":109,"threads_started":1}
I20260621 07:51:25.520000 22642 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2d109f8bd2c4300b64e736c837f3871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d109f8bd2c4300b64e736c837f3871" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 37763 } }
I20260621 07:51:25.520192 22642 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2d109f8bd2c4300b64e736c837f3871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:25.520258 22642 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2d109f8bd2c4300b64e736c837f3871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2d109f8bd2c4300b64e736c837f3871, State: Initialized, Role: FOLLOWER
I20260621 07:51:25.520610 771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:51:25.520673 22642 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2d109f8bd2c4300b64e736c837f3871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d109f8bd2c4300b64e736c837f3871" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 37763 } }
I20260621 07:51:25.520803 22642 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2d109f8bd2c4300b64e736c837f3871 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:51:25.520851 22642 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2d109f8bd2c4300b64e736c837f3871 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:51:25.520925 22642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2d109f8bd2c4300b64e736c837f3871 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:25.521816 22642 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2d109f8bd2c4300b64e736c837f3871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d109f8bd2c4300b64e736c837f3871" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 37763 } }
I20260621 07:51:25.522133 22642 leader_election.cc:302] T 00000000000000000000000000000000 P c2d109f8bd2c4300b64e736c837f3871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2d109f8bd2c4300b64e736c837f3871; no voters:
I20260621 07:51:25.522421 22642 leader_election.cc:288] T 00000000000000000000000000000000 P c2d109f8bd2c4300b64e736c837f3871 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:51:25.522493 22651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2d109f8bd2c4300b64e736c837f3871 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:25.522683 22651 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2d109f8bd2c4300b64e736c837f3871 [term 1 LEADER]: Becoming Leader. State: Replica: c2d109f8bd2c4300b64e736c837f3871, State: Running, Role: LEADER
I20260621 07:51:25.522939 22651 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2d109f8bd2c4300b64e736c837f3871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d109f8bd2c4300b64e736c837f3871" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 37763 } }
I20260621 07:51:25.523520 22642 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2d109f8bd2c4300b64e736c837f3871 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:51:25.523993 22651 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2d109f8bd2c4300b64e736c837f3871 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2d109f8bd2c4300b64e736c837f3871. Latest consensus state: current_term: 1 leader_uuid: "c2d109f8bd2c4300b64e736c837f3871" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d109f8bd2c4300b64e736c837f3871" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 37763 } } }
I20260621 07:51:25.524142 22651 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2d109f8bd2c4300b64e736c837f3871 [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:25.524418 22645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:25.522570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42527 (local address 127.0.192.254:37763)
0621 07:51:25.522660 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:25.522663 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:25.522931 (+ 268us) server_negotiation.cc:407] Connection header received
0621 07:51:25.523033 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:25.523036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:25.523063 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:25.523121 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:25.523813 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:25.523816 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:25.523818 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:51:25.523830 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:25.523858 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:51:25.523871 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:25.523944 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:25.524292 (+ 348us) server_negotiation.cc:299] Negotiation successful
0621 07:51:25.524324 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260621 07:51:25.524384 22651 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2d109f8bd2c4300b64e736c837f3871 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2d109f8bd2c4300b64e736c837f3871" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d109f8bd2c4300b64e736c837f3871" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 37763 } } }
I20260621 07:51:25.524488 22651 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2d109f8bd2c4300b64e736c837f3871 [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:25.524771 22659 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:51:25.527216 22659 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:51:25.531859 22659 catalog_manager.cc:1347] Generated new cluster ID: 0bf8b4221e76402abc7cf741f91e24e4
I20260621 07:51:25.531915 22659 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:51:25.540527 22659 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:51:25.540813 22659 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:51:25.544936 22659 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2d109f8bd2c4300b64e736c837f3871: Generated new TSK 0
I20260621 07:51:25.545470 22659 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:51:25.551455 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 22574
2026-06-21T07:51:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1238 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028028522580-771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:51:25 dist-test-slave-lzx3 krb5kdc[22673](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:51:25 dist-test-slave-lzx3 krb5kdc[22673](info): set up 2 sockets
Jun 21 07:51:25 dist-test-slave-lzx3 krb5kdc[22673](info): commencing operation
krb5kdc: starting...
W20260621 07:51:26.512671 771 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028028522580-771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028028522580-771-0/krb5kdc/test-user.keytab.
Jun 21 07:51:26 dist-test-slave-lzx3 krb5kdc[22673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:51:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:51:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.192.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
I20260621 07:51:26.618649 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:39597
--webserver_interface=127.0.192.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:40789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:39597
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:26.713140 22686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:26.713366 22686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:26.713420 22686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:26.717247 22686 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:51:26.717298 22686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:26.717315 22686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:26.717329 22686 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:51:26.717342 22686 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:51:26.722263 22686 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:40789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:39597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:39597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:26.723141 22686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:26.723997 22686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:26.731742 22691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:26.731757 22692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:26.731775 22694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:26.732076 22686 server_base.cc:1034] running on GCE node
I20260621 07:51:26.732678 22686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:26.733659 22686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:26.734822 22686 hybrid_clock.cc:648] HybridClock initialized: now 1782028286734796 us; error 42 us; skew 500 ppm
Jun 21 07:51:26 dist-test-slave-lzx3 krb5kdc[22673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:26.737584 22686 init.cc:377] Logged in from keytab as kudu/127.0.192.254@KRBTEST.COM (short username kudu)
I20260621 07:51:26.738478 22686 webserver.cc:460] Webserver started at http://127.0.192.254:40883/ using document root <none> and password file <none>
I20260621 07:51:26.739003 22686 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:26.739089 22686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:26.739312 22686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:26.741297 22686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "7ad360b7c7324434b2e9ae5f9fbbe58f"
format_stamp: "Formatted at 2026-06-21 07:51:26 on dist-test-slave-lzx3"
server_key: "a3212027ee1875340df4a3c207d68aec"
server_key_iv: "b6e860e913d11cbe33d77bfaed3c4c8b"
server_key_version: "encryptionkey@0"
I20260621 07:51:26.741803 22686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "7ad360b7c7324434b2e9ae5f9fbbe58f"
format_stamp: "Formatted at 2026-06-21 07:51:26 on dist-test-slave-lzx3"
server_key: "a3212027ee1875340df4a3c207d68aec"
server_key_iv: "b6e860e913d11cbe33d77bfaed3c4c8b"
server_key_version: "encryptionkey@0"
I20260621 07:51:26.745087 22686 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:51:26.747104 22701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:26.748157 22686 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:51:26.748298 22686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "7ad360b7c7324434b2e9ae5f9fbbe58f"
format_stamp: "Formatted at 2026-06-21 07:51:26 on dist-test-slave-lzx3"
server_key: "a3212027ee1875340df4a3c207d68aec"
server_key_iv: "b6e860e913d11cbe33d77bfaed3c4c8b"
server_key_version: "encryptionkey@0"
I20260621 07:51:26.748428 22686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:26.772401 22686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:26.775046 22686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:26.775225 22686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:26.781765 22686 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:39597
I20260621 07:51:26.781813 22753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:39597 every 8 connection(s)
I20260621 07:51:26.782554 22686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:51:26.783959 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 22686
I20260621 07:51:26.784085 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:51:26.784374 771 external_mini_cluster.cc:1428] Setting key 890b0a0dc4325f1e27de89e82dfca0c6
I20260621 07:51:26.785331 22754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:26.791065 22754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ad360b7c7324434b2e9ae5f9fbbe58f: Bootstrap starting.
Jun 21 07:51:26 dist-test-slave-lzx3 krb5kdc[22673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:51:26.793406 22754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ad360b7c7324434b2e9ae5f9fbbe58f: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:26.794162 22754 log.cc:824] T 00000000000000000000000000000000 P 7ad360b7c7324434b2e9ae5f9fbbe58f: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:26.796159 22754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ad360b7c7324434b2e9ae5f9fbbe58f: No bootstrap required, opened a new log
I20260621 07:51:26.797425 22757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:26.785544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46307 (local address 127.0.192.254:39597)
0621 07:51:26.785899 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:26.785919 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:26.785950 (+ 31us) server_negotiation.cc:407] Connection header received
0621 07:51:26.786522 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:26.786536 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:26.786825 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:26.787063 (+ 238us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:26.787890 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:26.788909 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:26.790035 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:26.790419 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:26.790519 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:26.792984 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:26.792989 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:26.793000 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:26.793024 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:26.794662 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:26.795072 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:26.795078 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:26.795082 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:26.795201 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:26.795454 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:26.795456 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:26.795458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:26.795923 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:26.796105 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:26.796304 (+ 199us) server_negotiation.cc:299] Negotiation successful
0621 07:51:26.796649 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":124,"threads_started":1}
I20260621 07:51:26.800042 771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:51:26.800067 22754 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ad360b7c7324434b2e9ae5f9fbbe58f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad360b7c7324434b2e9ae5f9fbbe58f" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 39597 } }
I20260621 07:51:26.800283 22754 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ad360b7c7324434b2e9ae5f9fbbe58f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:26.800357 22754 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ad360b7c7324434b2e9ae5f9fbbe58f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ad360b7c7324434b2e9ae5f9fbbe58f, State: Initialized, Role: FOLLOWER
I20260621 07:51:26.800753 22754 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ad360b7c7324434b2e9ae5f9fbbe58f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad360b7c7324434b2e9ae5f9fbbe58f" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 39597 } }
I20260621 07:51:26.800879 22754 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ad360b7c7324434b2e9ae5f9fbbe58f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:51:26.800956 22754 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ad360b7c7324434b2e9ae5f9fbbe58f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:51:26.801039 22754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ad360b7c7324434b2e9ae5f9fbbe58f [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:26.801867 22754 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ad360b7c7324434b2e9ae5f9fbbe58f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad360b7c7324434b2e9ae5f9fbbe58f" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 39597 } }
I20260621 07:51:26.802212 22754 leader_election.cc:302] T 00000000000000000000000000000000 P 7ad360b7c7324434b2e9ae5f9fbbe58f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ad360b7c7324434b2e9ae5f9fbbe58f; no voters:
I20260621 07:51:26.802521 22754 leader_election.cc:288] T 00000000000000000000000000000000 P 7ad360b7c7324434b2e9ae5f9fbbe58f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:51:26.802594 22764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ad360b7c7324434b2e9ae5f9fbbe58f [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:26.802774 22764 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ad360b7c7324434b2e9ae5f9fbbe58f [term 1 LEADER]: Becoming Leader. State: Replica: 7ad360b7c7324434b2e9ae5f9fbbe58f, State: Running, Role: LEADER
I20260621 07:51:26.803125 22764 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ad360b7c7324434b2e9ae5f9fbbe58f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad360b7c7324434b2e9ae5f9fbbe58f" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 39597 } }
I20260621 07:51:26.803591 22754 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ad360b7c7324434b2e9ae5f9fbbe58f [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:51:26.804440 22765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ad360b7c7324434b2e9ae5f9fbbe58f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ad360b7c7324434b2e9ae5f9fbbe58f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad360b7c7324434b2e9ae5f9fbbe58f" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 39597 } } }
I20260621 07:51:26.804569 22765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ad360b7c7324434b2e9ae5f9fbbe58f [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:26.804900 22766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ad360b7c7324434b2e9ae5f9fbbe58f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ad360b7c7324434b2e9ae5f9fbbe58f. Latest consensus state: current_term: 1 leader_uuid: "7ad360b7c7324434b2e9ae5f9fbbe58f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad360b7c7324434b2e9ae5f9fbbe58f" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 39597 } } }
I20260621 07:51:26.805027 22766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ad360b7c7324434b2e9ae5f9fbbe58f [sys.catalog]: This master's current role is: LEADER
W20260621 07:51:26.808197 22780 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7ad360b7c7324434b2e9ae5f9fbbe58f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:51:26.808336 22780 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:51:26.808408 22773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:51:26.809288 22773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:51:26.809598 22757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:26.802106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40211 (local address 127.0.192.254:39597)
0621 07:51:26.802194 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:26.802197 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:26.802330 (+ 133us) server_negotiation.cc:407] Connection header received
0621 07:51:26.802463 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:26.802465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:26.802540 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:26.802615 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:26.803357 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:26.804000 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:26.805117 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:26.805355 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:26.805457 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:26.806755 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:26.806758 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:26.806759 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:26.806784 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:26.807956 (+ 1172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:26.808348 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:26.808351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:26.808353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:26.808489 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:26.808797 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:26.808800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:26.808802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:26.808955 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:26.809082 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:26.809312 (+ 230us) server_negotiation.cc:299] Negotiation successful
0621 07:51:26.809422 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260621 07:51:26.814213 22773 catalog_manager.cc:1347] Generated new cluster ID: 7aa09f3465ec459f9ebe64ec304c3e1b
I20260621 07:51:26.814287 22773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:51:26.819419 22773 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:51:26.820142 22773 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:51:26.826488 22773 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ad360b7c7324434b2e9ae5f9fbbe58f: Generated new TSK 0
I20260621 07:51:26.827093 22773 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:51:26.836025 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 22686
2026-06-21T07:51:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1284 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028028522580-771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:51:26 dist-test-slave-lzx3 krb5kdc[22785](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:51:26 dist-test-slave-lzx3 krb5kdc[22785](info): set up 2 sockets
Jun 21 07:51:26 dist-test-slave-lzx3 krb5kdc[22785](info): commencing operation
krb5kdc: starting...
W20260621 07:51:27.808656 771 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028028522580-771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028028522580-771-0/krb5kdc/test-user.keytab.
Jun 21 07:51:27 dist-test-slave-lzx3 krb5kdc[22785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:51:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:51:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.192.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
I20260621 07:51:27.917323 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:39931
--webserver_interface=127.0.192.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:39215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:39931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:28.013506 22798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:28.013764 22798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:28.013829 22798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:28.017674 22798 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:51:28.017762 22798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:28.017802 22798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:28.017836 22798 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:51:28.017867 22798 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:51:28.022507 22798 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:39215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:39931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:39931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:28.023419 22798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:28.024326 22798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:28.031742 22804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:28.031885 22806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:28.031713 22803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:28.031991 22798 server_base.cc:1034] running on GCE node
I20260621 07:51:28.032629 22798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:28.033532 22798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:28.034706 22798 hybrid_clock.cc:648] HybridClock initialized: now 1782028288034682 us; error 34 us; skew 500 ppm
Jun 21 07:51:28 dist-test-slave-lzx3 krb5kdc[22785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:28.037264 22798 init.cc:377] Logged in from keytab as kudu/127.0.192.254@KRBTEST.COM (short username kudu)
I20260621 07:51:28.037953 22798 webserver.cc:460] Webserver started at http://127.0.192.254:46801/ using document root <none> and password file <none>
I20260621 07:51:28.038443 22798 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:28.038499 22798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:28.038645 22798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:28.040298 22798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "6554890c216642e3b793de4ca7b76137"
format_stamp: "Formatted at 2026-06-21 07:51:28 on dist-test-slave-lzx3"
server_key: "b5d67094fd42f5841d001dd083928159"
server_key_iv: "2e7e966ac7fdd95fb8ed87923cb13fab"
server_key_version: "encryptionkey@0"
I20260621 07:51:28.040844 22798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "6554890c216642e3b793de4ca7b76137"
format_stamp: "Formatted at 2026-06-21 07:51:28 on dist-test-slave-lzx3"
server_key: "b5d67094fd42f5841d001dd083928159"
server_key_iv: "2e7e966ac7fdd95fb8ed87923cb13fab"
server_key_version: "encryptionkey@0"
I20260621 07:51:28.044056 22798 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260621 07:51:28.046176 22813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:28.047166 22798 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 07:51:28.047276 22798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "6554890c216642e3b793de4ca7b76137"
format_stamp: "Formatted at 2026-06-21 07:51:28 on dist-test-slave-lzx3"
server_key: "b5d67094fd42f5841d001dd083928159"
server_key_iv: "2e7e966ac7fdd95fb8ed87923cb13fab"
server_key_version: "encryptionkey@0"
I20260621 07:51:28.047367 22798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:28.065739 22798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:28.068593 22798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:28.068781 22798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:28.075412 22798 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:39931
I20260621 07:51:28.075428 22865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:39931 every 8 connection(s)
I20260621 07:51:28.076324 22798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:51:28.078755 22866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:28.082259 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 22798
I20260621 07:51:28.082404 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:51:28.082664 771 external_mini_cluster.cc:1428] Setting key 9ffc5abed768dfae372a37faa9b8ab73
I20260621 07:51:28.083770 22866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6554890c216642e3b793de4ca7b76137: Bootstrap starting.
I20260621 07:51:28.085707 22866 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6554890c216642e3b793de4ca7b76137: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:28.086316 22866 log.cc:824] T 00000000000000000000000000000000 P 6554890c216642e3b793de4ca7b76137: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:28.088138 22866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6554890c216642e3b793de4ca7b76137: No bootstrap required, opened a new log
I20260621 07:51:28.090826 22869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:28.083844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48959 (local address 127.0.192.254:39931)
0621 07:51:28.084209 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:28.084222 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:28.084253 (+ 31us) server_negotiation.cc:407] Connection header received
0621 07:51:28.084877 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:28.084893 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:28.085174 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:28.085460 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:28.085851 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:28.086800 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:28.087869 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:28.088383 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:28.088498 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:28.089314 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:28.089319 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:28.089329 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:51:28.089679 (+ 350us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:28.089724 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:51:28.089744 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:28.089831 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:28.089971 (+ 140us) server_negotiation.cc:299] Negotiation successful
0621 07:51:28.090177 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":100,"threads_started":1}
I20260621 07:51:28.091862 22866 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6554890c216642e3b793de4ca7b76137 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6554890c216642e3b793de4ca7b76137" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 39931 } }
I20260621 07:51:28.092083 22866 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6554890c216642e3b793de4ca7b76137 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:28.092132 22866 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6554890c216642e3b793de4ca7b76137 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6554890c216642e3b793de4ca7b76137, State: Initialized, Role: FOLLOWER
I20260621 07:51:28.092506 22866 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6554890c216642e3b793de4ca7b76137 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6554890c216642e3b793de4ca7b76137" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 39931 } }
I20260621 07:51:28.092742 22866 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6554890c216642e3b793de4ca7b76137 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:51:28.092813 22866 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6554890c216642e3b793de4ca7b76137 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:51:28.092914 22866 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6554890c216642e3b793de4ca7b76137 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:28.093497 771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:51:28.093834 22866 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6554890c216642e3b793de4ca7b76137 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6554890c216642e3b793de4ca7b76137" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 39931 } }
I20260621 07:51:28.094133 22866 leader_election.cc:302] T 00000000000000000000000000000000 P 6554890c216642e3b793de4ca7b76137 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6554890c216642e3b793de4ca7b76137; no voters:
I20260621 07:51:28.094405 22866 leader_election.cc:288] T 00000000000000000000000000000000 P 6554890c216642e3b793de4ca7b76137 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:51:28.094458 22871 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6554890c216642e3b793de4ca7b76137 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:28.094623 22871 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6554890c216642e3b793de4ca7b76137 [term 1 LEADER]: Becoming Leader. State: Replica: 6554890c216642e3b793de4ca7b76137, State: Running, Role: LEADER
I20260621 07:51:28.094925 22871 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6554890c216642e3b793de4ca7b76137 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6554890c216642e3b793de4ca7b76137" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 39931 } }
I20260621 07:51:28.095530 22866 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6554890c216642e3b793de4ca7b76137 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:51:28.096012 22871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6554890c216642e3b793de4ca7b76137 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6554890c216642e3b793de4ca7b76137. Latest consensus state: current_term: 1 leader_uuid: "6554890c216642e3b793de4ca7b76137" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6554890c216642e3b793de4ca7b76137" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 39931 } } }
I20260621 07:51:28.096122 22871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6554890c216642e3b793de4ca7b76137 [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:28.096338 22871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6554890c216642e3b793de4ca7b76137 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6554890c216642e3b793de4ca7b76137" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6554890c216642e3b793de4ca7b76137" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 39931 } } }
I20260621 07:51:28.096454 22871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6554890c216642e3b793de4ca7b76137 [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:28.099637 22869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:28.095400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43295 (local address 127.0.192.254:39931)
0621 07:51:28.095640 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:28.095646 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:28.095660 (+ 14us) server_negotiation.cc:407] Connection header received
0621 07:51:28.095809 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:28.095812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:28.095852 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:28.095921 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:28.096316 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:28.097068 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:28.098224 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:28.098435 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:28.098513 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:28.099235 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:28.099238 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:28.099240 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:51:28.099252 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:28.099275 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:51:28.099287 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:28.099375 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:28.099472 (+ 97us) server_negotiation.cc:299] Negotiation successful
0621 07:51:28.099521 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
W20260621 07:51:28.100754 22892 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6554890c216642e3b793de4ca7b76137: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:51:28.100838 22892 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:51:28.100908 22880 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:51:28.101511 22880 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:51:28.106011 22880 catalog_manager.cc:1347] Generated new cluster ID: 2428550a78b643518c778a6689cee6c8
I20260621 07:51:28.106066 22880 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:51:28.139071 22880 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:51:28.139963 22880 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:51:28.144570 22880 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6554890c216642e3b793de4ca7b76137: Generated new TSK 0
I20260621 07:51:28.145176 22880 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:51:28.155202 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 22798
2026-06-21T07:51:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1318 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028028522580-771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:51:28 dist-test-slave-lzx3 krb5kdc[22897](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:51:28 dist-test-slave-lzx3 krb5kdc[22897](info): set up 2 sockets
Jun 21 07:51:28 dist-test-slave-lzx3 krb5kdc[22897](info): commencing operation
krb5kdc: starting...
W20260621 07:51:29.108767 771 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028028522580-771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028028522580-771-0/krb5kdc/test-user.keytab.
Jun 21 07:51:29 dist-test-slave-lzx3 krb5kdc[22897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:51:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:51:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.192.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
I20260621 07:51:29.213840 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:34597
--webserver_interface=127.0.192.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:38699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:34597
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:29.309435 22910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:29.309648 22910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:29.309707 22910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:29.313482 22910 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:51:29.313547 22910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:29.313580 22910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:29.313609 22910 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:51:29.313640 22910 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:51:29.318216 22910 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:38699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:34597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:34597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:29.319074 22910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:29.319834 22910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:29.326815 22918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:29.326805 22916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:29.327023 22910 server_base.cc:1034] running on GCE node
W20260621 07:51:29.326805 22915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:29.327744 22910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:29.328787 22910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:29.329989 22910 hybrid_clock.cc:648] HybridClock initialized: now 1782028289329975 us; error 28 us; skew 500 ppm
Jun 21 07:51:29 dist-test-slave-lzx3 krb5kdc[22897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:29.332284 22910 init.cc:377] Logged in from keytab as kudu/127.0.192.254@KRBTEST.COM (short username kudu)
I20260621 07:51:29.333015 22910 webserver.cc:460] Webserver started at http://127.0.192.254:43163/ using document root <none> and password file <none>
I20260621 07:51:29.333405 22910 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:29.333458 22910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:29.333585 22910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:29.335172 22910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "49b559b71faf4f56bd34d04debdd1827"
format_stamp: "Formatted at 2026-06-21 07:51:29 on dist-test-slave-lzx3"
server_key: "988c6fe3fee322d52d69ec2f19ca5225"
server_key_iv: "2529733eb86f3967315debfc036eae5a"
server_key_version: "encryptionkey@0"
I20260621 07:51:29.335633 22910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "49b559b71faf4f56bd34d04debdd1827"
format_stamp: "Formatted at 2026-06-21 07:51:29 on dist-test-slave-lzx3"
server_key: "988c6fe3fee322d52d69ec2f19ca5225"
server_key_iv: "2529733eb86f3967315debfc036eae5a"
server_key_version: "encryptionkey@0"
I20260621 07:51:29.338730 22910 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 07:51:29.340581 22925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:29.341519 22910 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:51:29.341625 22910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "49b559b71faf4f56bd34d04debdd1827"
format_stamp: "Formatted at 2026-06-21 07:51:29 on dist-test-slave-lzx3"
server_key: "988c6fe3fee322d52d69ec2f19ca5225"
server_key_iv: "2529733eb86f3967315debfc036eae5a"
server_key_version: "encryptionkey@0"
I20260621 07:51:29.341709 22910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:29.345273 22910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:29.347397 22910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:29.347571 22910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:29.353601 22910 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:34597
I20260621 07:51:29.353615 22977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:34597 every 8 connection(s)
I20260621 07:51:29.354370 22910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:51:29.356684 22978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:29.358568 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 22910
I20260621 07:51:29.358713 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:51:29.359005 771 external_mini_cluster.cc:1428] Setting key b2a645c9d4c908ff0743c60533e0780f
I20260621 07:51:29.361769 22978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49b559b71faf4f56bd34d04debdd1827: Bootstrap starting.
I20260621 07:51:29.363340 22981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:29.360211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53527 (local address 127.0.192.254:34597)
0621 07:51:29.360618 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:29.360632 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:29.360668 (+ 36us) server_negotiation.cc:407] Connection header received
0621 07:51:29.361207 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:29.361220 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:29.361448 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:29.361676 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:29.362344 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:29.362348 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:29.362358 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:51:29.362365 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:29.362403 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:51:29.362417 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:29.362505 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:29.362610 (+ 105us) server_negotiation.cc:299] Negotiation successful
0621 07:51:29.362748 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":138,"threads_started":1}
I20260621 07:51:29.364042 22978 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49b559b71faf4f56bd34d04debdd1827: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:29.364734 22978 log.cc:824] T 00000000000000000000000000000000 P 49b559b71faf4f56bd34d04debdd1827: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:29.365605 771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:51:29.366287 22978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49b559b71faf4f56bd34d04debdd1827: No bootstrap required, opened a new log
I20260621 07:51:29.368775 22981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:29.367319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39521 (local address 127.0.192.254:34597)
0621 07:51:29.367420 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:29.367425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:29.367556 (+ 131us) server_negotiation.cc:407] Connection header received
0621 07:51:29.367662 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:29.367664 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:29.367710 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:29.367770 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:29.368377 (+ 607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:29.368380 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:29.368381 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:51:29.368391 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:29.368412 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:51:29.368422 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:29.368488 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:29.368612 (+ 124us) server_negotiation.cc:299] Negotiation successful
0621 07:51:29.368645 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260621 07:51:29.369995 22978 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49b559b71faf4f56bd34d04debdd1827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49b559b71faf4f56bd34d04debdd1827" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34597 } }
I20260621 07:51:29.370178 22978 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49b559b71faf4f56bd34d04debdd1827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:51:29.370226 22978 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49b559b71faf4f56bd34d04debdd1827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49b559b71faf4f56bd34d04debdd1827, State: Initialized, Role: FOLLOWER
I20260621 07:51:29.370594 22978 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49b559b71faf4f56bd34d04debdd1827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49b559b71faf4f56bd34d04debdd1827" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34597 } }
I20260621 07:51:29.370726 22978 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49b559b71faf4f56bd34d04debdd1827 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:51:29.370784 22978 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49b559b71faf4f56bd34d04debdd1827 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:51:29.370864 22978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49b559b71faf4f56bd34d04debdd1827 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:29.371587 22978 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49b559b71faf4f56bd34d04debdd1827 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49b559b71faf4f56bd34d04debdd1827" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34597 } }
I20260621 07:51:29.371861 22978 leader_election.cc:302] T 00000000000000000000000000000000 P 49b559b71faf4f56bd34d04debdd1827 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49b559b71faf4f56bd34d04debdd1827; no voters:
I20260621 07:51:29.372133 22978 leader_election.cc:288] T 00000000000000000000000000000000 P 49b559b71faf4f56bd34d04debdd1827 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:51:29.372203 22988 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49b559b71faf4f56bd34d04debdd1827 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:29.372386 22988 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49b559b71faf4f56bd34d04debdd1827 [term 1 LEADER]: Becoming Leader. State: Replica: 49b559b71faf4f56bd34d04debdd1827, State: Running, Role: LEADER
I20260621 07:51:29.372676 22988 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49b559b71faf4f56bd34d04debdd1827 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49b559b71faf4f56bd34d04debdd1827" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34597 } }
I20260621 07:51:29.373193 22978 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49b559b71faf4f56bd34d04debdd1827 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:51:29.374099 22990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49b559b71faf4f56bd34d04debdd1827 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49b559b71faf4f56bd34d04debdd1827" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49b559b71faf4f56bd34d04debdd1827" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34597 } } }
I20260621 07:51:29.374210 22990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49b559b71faf4f56bd34d04debdd1827 [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:29.374480 22989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49b559b71faf4f56bd34d04debdd1827 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49b559b71faf4f56bd34d04debdd1827. Latest consensus state: current_term: 1 leader_uuid: "49b559b71faf4f56bd34d04debdd1827" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49b559b71faf4f56bd34d04debdd1827" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34597 } } }
I20260621 07:51:29.374543 22997 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:51:29.374650 22989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49b559b71faf4f56bd34d04debdd1827 [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:29.377390 22997 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:51:29.381712 22997 catalog_manager.cc:1347] Generated new cluster ID: 81164e91013a4abc9e30c94c6ceb9c2f
I20260621 07:51:29.381767 22997 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:51:29.395272 22997 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:51:29.395514 22997 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:51:29.398241 22997 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49b559b71faf4f56bd34d04debdd1827: Generated new TSK 0
I20260621 07:51:29.398739 22997 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:51:29.401681 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 22910
2026-06-21T07:51:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1246 ms)
[----------] 18 tests from AuthTokenIssuingTest (22914 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:51:29 dist-test-slave-lzx3 krb5kdc[23009](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:51:29 dist-test-slave-lzx3 krb5kdc[23009](info): set up 2 sockets
Jun 21 07:51:29 dist-test-slave-lzx3 krb5kdc[23009](info): commencing operation
krb5kdc: starting...
W20260621 07:51:30.352540 771 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/krb5kdc/test-user.keytab.
Jun 21 07:51:30 dist-test-slave-lzx3 krb5kdc[23009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:51:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:51:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.192.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
I20260621 07:51:30.455878 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:37879
--webserver_interface=127.0.192.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:42679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:37879,127.0.192.253:46609,127.0.192.252:42441
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:30.551640 23022 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 07:51:30.551946 23022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:30.552022 23022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:30.552091 23022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:30.555806 23022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:51:30.555891 23022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:30.555927 23022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:30.555960 23022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:51:30.555989 23022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:51:30.560806 23022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:42679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:37879,127.0.192.253:46609,127.0.192.252:42441
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:37879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:30.561719 23022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:30.562613 23022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:30.570037 23027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:30.570065 23028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:30.570066 23030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:30.570206 23022 server_base.cc:1034] running on GCE node
I20260621 07:51:30.570796 23022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:30.571720 23022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:30.572866 23022 hybrid_clock.cc:648] HybridClock initialized: now 1782028290572841 us; error 37 us; skew 500 ppm
Jun 21 07:51:30 dist-test-slave-lzx3 krb5kdc[23009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028290, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:30.579048 23022 init.cc:377] Logged in from keytab as kudu/127.0.192.254@KRBTEST.COM (short username kudu)
I20260621 07:51:30.579731 23022 webserver.cc:460] Webserver started at http://127.0.192.254:43603/ using document root <none> and password file <none>
I20260621 07:51:30.580231 23022 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:30.580312 23022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:30.580511 23022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:30.582295 23022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "ec761274c9224942a6f2a02bb9cff3f7"
format_stamp: "Formatted at 2026-06-21 07:51:30 on dist-test-slave-lzx3"
server_key: "bcd1a199e359c3e4dc69ba4ee2deacb3"
server_key_iv: "7d5b71dce1d855673420071a5a58e3e8"
server_key_version: "encryptionkey@0"
I20260621 07:51:30.582763 23022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "ec761274c9224942a6f2a02bb9cff3f7"
format_stamp: "Formatted at 2026-06-21 07:51:30 on dist-test-slave-lzx3"
server_key: "bcd1a199e359c3e4dc69ba4ee2deacb3"
server_key_iv: "7d5b71dce1d855673420071a5a58e3e8"
server_key_version: "encryptionkey@0"
I20260621 07:51:30.586024 23022 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260621 07:51:30.588193 23037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:30.589396 23022 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:51:30.589536 23022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "ec761274c9224942a6f2a02bb9cff3f7"
format_stamp: "Formatted at 2026-06-21 07:51:30 on dist-test-slave-lzx3"
server_key: "bcd1a199e359c3e4dc69ba4ee2deacb3"
server_key_iv: "7d5b71dce1d855673420071a5a58e3e8"
server_key_version: "encryptionkey@0"
I20260621 07:51:30.589629 23022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:30.598584 23022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:30.600726 23022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:30.600879 23022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:30.606662 23022 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:37879
I20260621 07:51:30.606882 23089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:37879 every 8 connection(s)
I20260621 07:51:30.607476 23022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:51:30.609910 23090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:30.610569 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 23022
I20260621 07:51:30.610695 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:51:30.610958 771 external_mini_cluster.cc:1428] Setting key 96fb8bb3c973e9cef6439064c8f48699
I20260621 07:51:30.613840 23090 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.192.254" port: 37879 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.192.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.253@KRBTEST.COM" created.
Jun 21 07:51:30 dist-test-slave-lzx3 krb5kdc[23009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028290, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:51:30.626922 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:30.615960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56768 (local address 127.0.192.254:37879)
0621 07:51:30.616297 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:30.616316 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:30.616591 (+ 275us) server_negotiation.cc:407] Connection header received
0621 07:51:30.617480 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:30.617484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:30.617644 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:30.617820 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:30.618588 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:30.619385 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:30.620246 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:30.620680 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:30.620834 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:30.622911 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:30.622915 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:30.622922 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:30.622940 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:30.624361 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:30.624901 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:30.624905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:30.624906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:30.624994 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:30.625306 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:30.625307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:30.625308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:30.625605 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:30.625702 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:30.626115 (+ 413us) server_negotiation.cc:299] Negotiation successful
0621 07:51:30.626282 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":221,"thread_start_us":118,"threads_started":1}
I20260621 07:51:30.626926 23094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:30.615961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:37879 (local address 127.0.0.1:56768)
0621 07:51:30.616446 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:30.616477 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:30.617190 (+ 713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:30.617818 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:30.617823 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:30.618231 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:30.618439 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:30.618450 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:30.619516 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:30.619525 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:30.620140 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:30.620147 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:30.620827 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:30.620830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:30.620912 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:30.621365 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:30.621377 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:30.622788 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:30.624464 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:30.624475 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:30.624484 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:30.624802 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:30.625089 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:30.625095 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:30.625098 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:30.625222 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:30.625723 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:30.625731 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:30.625881 (+ 150us) client_negotiation.cc:769] Sending connection context
0621 07:51:30.626067 (+ 186us) client_negotiation.cc:240] Negotiation successful
0621 07:51:30.626340 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":27,"spinlock_wait_cycles":22400,"thread_start_us":133,"threads_started":1}
I20260621 07:51:30.628942 23090 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.192.253" port: 46609 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:51:30.629494 23094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:30.629307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.253:46609 (local address 127.0.0.1:41232)
0621 07:51:30.629398 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:30.629451 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.253:46609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260621 07:51:30.629685 23041 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.192.253:46609: connect: Connection refused (error 111)
I20260621 07:51:30.630340 23094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:30.630129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.253:46609 (local address 127.0.0.1:41236)
0621 07:51:30.630244 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:30.630300 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.253:46609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260621 07:51:30.630474 23090 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.192.253:46609: Network error: Client connection negotiation failed: client connection to 127.0.192.253:46609: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/krb5kdc/kudu_127.0.192.253.keytab.
Entry for principal kudu/127.0.192.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/krb5kdc/kudu_127.0.192.253.keytab.
I20260621 07:51:30.633304 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.253:46609
--webserver_interface=127.0.192.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/krb5kdc/kudu_127.0.192.253.keytab
--principal=kudu/127.0.192.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:42679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:37879,127.0.192.253:46609,127.0.192.252:42441
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:51:30.679738 23090 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.192.253" port: 46609 } attempt: 1
I20260621 07:51:30.680433 23094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:30.680214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.253:46609 (local address 127.0.0.1:41246)
0621 07:51:30.680330 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:30.680378 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.253:46609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260621 07:51:30.681046 23094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:30.680881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.253:46609 (local address 127.0.0.1:41250)
0621 07:51:30.680964 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:30.681001 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.253:46609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260621 07:51:30.681187 23090 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.192.253:46609: Network error: Client connection negotiation failed: client connection to 127.0.192.253:46609: connect: Connection refused (error 111)
W20260621 07:51:30.728928 23097 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 07:51:30.729187 23097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:30.729259 23097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:30.729321 23097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:30.732615 23097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:51:30.732686 23097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:30.732717 23097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:30.732749 23097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:51:30.732777 23097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:51:30.737221 23097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:42679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:37879,127.0.192.253:46609,127.0.192.252:42441
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/krb5kdc/kudu_127.0.192.253.keytab
--principal=kudu/127.0.192.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.253:46609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.192.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:30.738143 23097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:30.739060 23097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:30.747022 23102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:30.747138 23097 server_base.cc:1034] running on GCE node
W20260621 07:51:30.747022 23105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:30.747063 23103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:30.747860 23097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:30.748929 23097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:30.749322 23090 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.192.253" port: 46609 } attempt: 2
I20260621 07:51:30.749991 23094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:30.749746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.253:46609 (local address 127.0.0.1:41258)
0621 07:51:30.749868 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:30.749913 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.253:46609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260621 07:51:30.750146 23097 hybrid_clock.cc:648] HybridClock initialized: now 1782028290750113 us; error 41 us; skew 500 ppm
I20260621 07:51:30.750761 23094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:30.750486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.253:46609 (local address 127.0.0.1:41264)
0621 07:51:30.750611 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:30.750673 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.253:46609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260621 07:51:30.750932 23090 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.192.253:46609: Network error: Client connection negotiation failed: client connection to 127.0.192.253:46609: connect: Connection refused (error 111)
Jun 21 07:51:30 dist-test-slave-lzx3 krb5kdc[23009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028290, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:30.752854 23097 init.cc:377] Logged in from keytab as kudu/127.0.192.253@KRBTEST.COM (short username kudu)
I20260621 07:51:30.753638 23097 webserver.cc:460] Webserver started at http://127.0.192.253:46785/ using document root <none> and password file <none>
I20260621 07:51:30.754174 23097 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:30.754264 23097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:30.754482 23097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:30.756467 23097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-1/data/instance:
uuid: "f94652df8a564b99b52490162c57aa9e"
format_stamp: "Formatted at 2026-06-21 07:51:30 on dist-test-slave-lzx3"
server_key: "dee2b7c78921c0050f42a003a3dc7c38"
server_key_iv: "ef69f1d90612129d8a5374b6e6049dcf"
server_key_version: "encryptionkey@0"
I20260621 07:51:30.757061 23097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-1/wal/instance:
uuid: "f94652df8a564b99b52490162c57aa9e"
format_stamp: "Formatted at 2026-06-21 07:51:30 on dist-test-slave-lzx3"
server_key: "dee2b7c78921c0050f42a003a3dc7c38"
server_key_iv: "ef69f1d90612129d8a5374b6e6049dcf"
server_key_version: "encryptionkey@0"
I20260621 07:51:30.760497 23097 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:51:30.762666 23112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:30.763746 23097 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260621 07:51:30.763882 23097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-1/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-1/wal
uuid: "f94652df8a564b99b52490162c57aa9e"
format_stamp: "Formatted at 2026-06-21 07:51:30 on dist-test-slave-lzx3"
server_key: "dee2b7c78921c0050f42a003a3dc7c38"
server_key_iv: "ef69f1d90612129d8a5374b6e6049dcf"
server_key_version: "encryptionkey@0"
I20260621 07:51:30.763993 23097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:30.779277 23097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:30.782078 23097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:30.782259 23097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:30.788403 23097 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.253:46609
I20260621 07:51:30.788712 23164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.253:46609 every 8 connection(s)
I20260621 07:51:30.789382 23097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-1/data/info.pb
I20260621 07:51:30.791872 23165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:30.795280 23165 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.192.254" port: 37879 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:51:30.798458 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 23097
I20260621 07:51:30.798602 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-1/wal/instance
I20260621 07:51:30.798916 771 external_mini_cluster.cc:1428] Setting key f4c89deda30bea2f25688a2989f65612
Jun 21 07:51:30 dist-test-slave-lzx3 krb5kdc[23009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028290, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.253@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.192.252@KRBTEST.COM; defaulting to no policy
I20260621 07:51:30.808053 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:30.796850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56774 (local address 127.0.192.254:37879)
0621 07:51:30.796984 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:30.796989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:30.797747 (+ 758us) server_negotiation.cc:407] Connection header received
0621 07:51:30.798567 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:30.798570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:30.798623 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:30.798736 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:30.800201 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:30.800751 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:30.801890 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:30.802138 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:30.802251 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:30.804523 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:30.804528 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:30.804532 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:30.804579 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:30.806207 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:30.806670 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:30.806673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:30.806675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:30.806760 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:30.807107 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:30.807111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:30.807113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:30.807285 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:30.807385 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:30.807798 (+ 413us) server_negotiation.cc:299] Negotiation successful
0621 07:51:30.807937 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 07:51:30.808878 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:30.797228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:37879 (local address 127.0.0.1:56774)
0621 07:51:30.797612 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:30.797649 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:30.798361 (+ 712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:30.798940 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:30.798951 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:30.799386 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:30.800060 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:30.800080 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:30.800898 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:30.800901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:30.801750 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:30.801759 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:30.802284 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:30.802288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:30.802451 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:30.803045 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:30.803060 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:30.804361 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:30.806319 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:30.806323 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:30.806333 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:30.806572 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:30.806880 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:30.806885 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:30.806887 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:30.807012 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:30.807392 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:30.807396 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:30.807595 (+ 199us) client_negotiation.cc:769] Sending connection context
0621 07:51:30.807732 (+ 137us) client_negotiation.cc:240] Negotiation successful
0621 07:51:30.808052 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":93,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.252@KRBTEST.COM" created.
I20260621 07:51:30.810333 23165 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.192.253" port: 46609 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 07:51:30 dist-test-slave-lzx3 krb5kdc[23009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028290, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.253@KRBTEST.COM for kudu/127.0.192.253@KRBTEST.COM
I20260621 07:51:30.818959 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:30.810842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.253:46609 (local address 127.0.0.1:41276)
0621 07:51:30.810985 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:30.811001 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:30.811080 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:30.811546 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:30.811551 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:30.811743 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:30.811824 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:30.811829 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:30.812719 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:30.812725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:30.813222 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:30.813229 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:30.813750 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:30.813752 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:30.813878 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:30.814365 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:30.814375 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:30.815725 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:30.817511 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:30.817514 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:30.817516 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:30.817691 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:30.818007 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:30.818010 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:30.818012 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:30.818089 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:30.818535 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:30.818538 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:30.818618 (+ 80us) client_negotiation.cc:769] Sending connection context
0621 07:51:30.818719 (+ 101us) client_negotiation.cc:240] Negotiation successful
0621 07:51:30.818820 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":30}
I20260621 07:51:30.819099 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:30.810939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41276 (local address 127.0.192.253:46609)
0621 07:51:30.811222 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:30.811232 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:30.811259 (+ 27us) server_negotiation.cc:407] Connection header received
0621 07:51:30.811291 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:30.811295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:30.811420 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:30.811517 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:30.811916 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:30.812608 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:30.813328 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:30.813651 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:30.813752 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:30.815847 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:30.815854 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:30.815862 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:30.815879 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:30.817398 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:30.817833 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:30.817837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:30.817838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:30.817903 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:30.818214 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:30.818217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:30.818218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:30.818415 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:30.818565 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:30.818797 (+ 232us) server_negotiation.cc:299] Negotiation successful
0621 07:51:30.818932 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":198,"thread_start_us":136,"threads_started":1}
I20260621 07:51:30.820710 23165 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42441 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:51:30.821338 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:30.821122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:42441 (local address 127.0.0.1:43540)
0621 07:51:30.821228 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:30.821296 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.252:42441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260621 07:51:30.821542 23115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.192.252:42441: connect: Connection refused (error 111)
I20260621 07:51:30.822240 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:30.822061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:42441 (local address 127.0.0.1:43546)
0621 07:51:30.822152 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:30.822191 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.252:42441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/krb5kdc/kudu_127.0.192.252.keytab.
Entry for principal kudu/127.0.192.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/krb5kdc/kudu_127.0.192.252.keytab.
W20260621 07:51:30.822394 23165 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.192.252:42441: Network error: Client connection negotiation failed: client connection to 127.0.192.252:42441: connect: Connection refused (error 111)
I20260621 07:51:30.823071 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.252:42441
--webserver_interface=127.0.192.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/krb5kdc/kudu_127.0.192.252.keytab
--principal=kudu/127.0.192.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:42679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:37879,127.0.192.253:46609,127.0.192.252:42441
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:51:30.842116 23090 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.192.253" port: 46609 } attempt: 3
Jun 21 07:51:30 dist-test-slave-lzx3 krb5kdc[23009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028290, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for kudu/127.0.192.253@KRBTEST.COM
I20260621 07:51:30.850421 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:30.842618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41290 (local address 127.0.192.253:46609)
0621 07:51:30.842723 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:30.842726 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:30.842739 (+ 13us) server_negotiation.cc:407] Connection header received
0621 07:51:30.842901 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:30.842905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:30.842964 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:30.843028 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:30.843480 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:30.843957 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:30.844995 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:30.845209 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:30.845282 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:30.847384 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:30.847387 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:30.847395 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:30.847415 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:30.848827 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:30.849271 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:30.849275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:30.849277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:30.849350 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:30.849654 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:30.849657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:30.849658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:30.849818 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:30.849923 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:30.850177 (+ 254us) server_negotiation.cc:299] Negotiation successful
0621 07:51:30.850280 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260621 07:51:30.850421 23094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:30.842572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.253:46609 (local address 127.0.0.1:41290)
0621 07:51:30.842689 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:30.842702 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:30.842779 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:30.843069 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:30.843072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:30.843243 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:30.843372 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:30.843378 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:30.844052 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:30.844055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:30.844893 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:30.844903 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:30.845312 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:30.845316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:30.845430 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:30.845862 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:30.845889 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:30.847247 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:30.848963 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:30.848967 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:30.848969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:30.849177 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:30.849448 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:30.849452 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:30.849454 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:30.849556 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:30.849919 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:30.849922 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:30.850020 (+ 98us) client_negotiation.cc:769] Sending connection context
0621 07:51:30.850127 (+ 107us) client_negotiation.cc:240] Negotiation successful
0621 07:51:30.850254 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260621 07:51:30.851505 23090 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42441 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:51:30.852540 23094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:30.852311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:42441 (local address 127.0.0.1:43550)
0621 07:51:30.852429 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:30.852479 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.252:42441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260621 07:51:30.853291 23094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:30.853061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:42441 (local address 127.0.0.1:43554)
0621 07:51:30.853205 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:30.853243 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.252:42441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260621 07:51:30.853464 23090 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.192.252:42441: Network error: Client connection negotiation failed: client connection to 127.0.192.252:42441: connect: Connection refused (error 111)
I20260621 07:51:30.871690 23165 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42441 } attempt: 1
I20260621 07:51:30.872251 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:30.872044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:42441 (local address 127.0.0.1:43564)
0621 07:51:30.872154 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:30.872199 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.252:42441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260621 07:51:30.872897 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:30.872714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:42441 (local address 127.0.0.1:43578)
0621 07:51:30.872805 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:30.872849 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.252:42441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260621 07:51:30.873008 23165 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.192.252:42441: Network error: Client connection negotiation failed: client connection to 127.0.192.252:42441: connect: Connection refused (error 111)
I20260621 07:51:30.884721 23090 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42441 } attempt: 1
I20260621 07:51:30.885202 23094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:30.885024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:42441 (local address 127.0.0.1:43586)
0621 07:51:30.885117 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:30.885156 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.252:42441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260621 07:51:30.885732 23094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:30.885561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:42441 (local address 127.0.0.1:43596)
0621 07:51:30.885648 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:30.885690 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.252:42441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260621 07:51:30.885835 23090 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.192.252:42441: Network error: Client connection negotiation failed: client connection to 127.0.192.252:42441: connect: Connection refused (error 111)
W20260621 07:51:30.923023 23172 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 07:51:30.923259 23172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:30.923344 23172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:30.923414 23172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:30.926874 23172 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:51:30.926949 23172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:30.927003 23172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:30.927039 23172 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:51:30.927069 23172 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:51:30.931576 23172 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:42679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:37879,127.0.192.253:46609,127.0.192.252:42441
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/krb5kdc/kudu_127.0.192.252.keytab
--principal=kudu/127.0.192.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.252:42441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.192.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:30.932331 23172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:30.933187 23172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:30.940672 23177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:30.940659 23178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:30.940716 23172 server_base.cc:1034] running on GCE node
W20260621 07:51:30.940675 23180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:30.941159 23165 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42441 } attempt: 2
I20260621 07:51:30.941423 23172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:30.941807 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:30.941560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:42441 (local address 127.0.0.1:43606)
0621 07:51:30.941677 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:30.941732 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.252:42441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260621 07:51:30.942410 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:30.942193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:42441 (local address 127.0.0.1:43614)
0621 07:51:30.942290 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:30.942336 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.252:42441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260621 07:51:30.942539 23165 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.192.252:42441: Network error: Client connection negotiation failed: client connection to 127.0.192.252:42441: connect: Connection refused (error 111)
I20260621 07:51:30.942572 23172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:30.943749 23172 hybrid_clock.cc:648] HybridClock initialized: now 1782028290943727 us; error 31 us; skew 500 ppm
Jun 21 07:51:30 dist-test-slave-lzx3 krb5kdc[23009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028290, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:30.946395 23172 init.cc:377] Logged in from keytab as kudu/127.0.192.252@KRBTEST.COM (short username kudu)
I20260621 07:51:30.947122 23172 webserver.cc:460] Webserver started at http://127.0.192.252:33459/ using document root <none> and password file <none>
I20260621 07:51:30.947691 23172 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:30.947750 23172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:30.947904 23172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:30.949684 23172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-2/data/instance:
uuid: "094f0fbfe0d84f018aba47c11847b821"
format_stamp: "Formatted at 2026-06-21 07:51:30 on dist-test-slave-lzx3"
server_key: "8318d831e31d49c8041c8da704fad683"
server_key_iv: "164828461dff06f96d8925af8cbd2406"
server_key_version: "encryptionkey@0"
I20260621 07:51:30.950153 23172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-2/wal/instance:
uuid: "094f0fbfe0d84f018aba47c11847b821"
format_stamp: "Formatted at 2026-06-21 07:51:30 on dist-test-slave-lzx3"
server_key: "8318d831e31d49c8041c8da704fad683"
server_key_iv: "164828461dff06f96d8925af8cbd2406"
server_key_version: "encryptionkey@0"
I20260621 07:51:30.953318 23172 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:51:30.955367 23187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:30.956394 23172 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:51:30.956518 23172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-2/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-2/wal
uuid: "094f0fbfe0d84f018aba47c11847b821"
format_stamp: "Formatted at 2026-06-21 07:51:30 on dist-test-slave-lzx3"
server_key: "8318d831e31d49c8041c8da704fad683"
server_key_iv: "164828461dff06f96d8925af8cbd2406"
server_key_version: "encryptionkey@0"
I20260621 07:51:30.956660 23172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:30.960978 23090 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42441 } attempt: 2
I20260621 07:51:30.961553 23094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:30.961325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:42441 (local address 127.0.0.1:43630)
0621 07:51:30.961448 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:30.961498 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.252:42441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260621 07:51:30.962205 23094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:30.962018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:42441 (local address 127.0.0.1:43632)
0621 07:51:30.962109 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:30.962149 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.252:42441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260621 07:51:30.962311 23090 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.192.252:42441: Network error: Client connection negotiation failed: client connection to 127.0.192.252:42441: connect: Connection refused (error 111)
I20260621 07:51:30.969471 23172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:30.972194 23172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:30.972324 23172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:30.978439 23172 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.252:42441
I20260621 07:51:30.978674 23239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.252:42441 every 8 connection(s)
I20260621 07:51:30.979226 23172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-2/data/info.pb
I20260621 07:51:30.981611 23240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:30.984714 23240 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.192.254" port: 37879 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:51:30.988116 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 23172
I20260621 07:51:30.988248 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782028028522580-771-0/minicluster-data/master-2/wal/instance
I20260621 07:51:30.988627 771 external_mini_cluster.cc:1428] Setting key a932f21bc93763e22e36a78d2ed0fca9
Jun 21 07:51:30 dist-test-slave-lzx3 krb5kdc[23009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028290, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.252@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
Jun 21 07:51:30 dist-test-slave-lzx3 krb5kdc[23009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.192.252@KRBTEST.COM
I20260621 07:51:30.997290 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:30.986147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56776 (local address 127.0.192.254:37879)
0621 07:51:30.986261 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:30.986265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:30.987051 (+ 786us) server_negotiation.cc:407] Connection header received
0621 07:51:30.987999 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:30.988005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:30.988060 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:30.988159 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:30.989510 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:30.989986 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:30.991250 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:30.991446 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:30.991556 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:30.993884 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:30.993887 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:30.993889 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:30.993908 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:30.995421 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:30.995900 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:30.995904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:30.995906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:30.995981 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:30.996289 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:30.996293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:30.996295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:30.996468 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:30.996595 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:30.997029 (+ 434us) server_negotiation.cc:299] Negotiation successful
0621 07:51:30.997134 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260621 07:51:30.998057 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:30.986549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:37879 (local address 127.0.0.1:56776)
0621 07:51:30.986919 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:30.986955 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:30.987806 (+ 851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:30.988365 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:30.988372 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:30.988804 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:30.989333 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:30.989353 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:30.990091 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:30.990094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:30.991124 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:30.991132 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:30.991577 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:30.991581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:30.991769 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:30.992293 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:30.992315 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:30.993753 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:30.995516 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:30.995523 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:30.995529 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:30.995793 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:30.996081 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:30.996088 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:30.996089 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:30.996199 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:30.996594 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:30.996599 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:30.996795 (+ 196us) client_negotiation.cc:769] Sending connection context
0621 07:51:30.996976 (+ 181us) client_negotiation.cc:240] Negotiation successful
0621 07:51:30.997231 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":85,"threads_started":1}
I20260621 07:51:30.998976 23243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:30.989302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43644 (local address 127.0.192.252:42441)
0621 07:51:30.989621 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:30.989631 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:30.989793 (+ 162us) server_negotiation.cc:407] Connection header received
0621 07:51:30.989935 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:30.989940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:30.990104 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:30.990276 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:30.990921 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:30.991841 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:30.992807 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:30.993146 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:30.993218 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:30.995446 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:30.995454 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:30.995468 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:30.995496 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:30.996998 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:30.997477 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:30.997485 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:30.997487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:30.997568 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:30.997864 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:30.997866 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:30.997875 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:30.998063 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:30.998276 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:30.998532 (+ 256us) server_negotiation.cc:299] Negotiation successful
0621 07:51:30.998809 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":135,"threads_started":1}
I20260621 07:51:30.999274 23240 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.192.253" port: 46609 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 07:51:30 dist-test-slave-lzx3 krb5kdc[23009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028290, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.252@KRBTEST.COM for kudu/127.0.192.253@KRBTEST.COM
Jun 21 07:51:31 dist-test-slave-lzx3 krb5kdc[23009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:51:31.006785 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:30.999646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.253:46609 (local address 127.0.0.1:41298)
0621 07:51:30.999722 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:30.999735 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:30.999821 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:31.000070 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:31.000072 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:31.000210 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:31.000280 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:31.000285 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:31.001149 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:31.001151 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:31.001616 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:31.001624 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:31.002014 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:31.002017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:31.002106 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:31.002516 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:31.002528 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:31.003737 (+ 1209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:31.005410 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:31.005416 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:31.005421 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:31.005637 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:31.005890 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:31.005892 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:31.005893 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:31.005959 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:31.006379 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:31.006381 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:31.006474 (+ 93us) client_negotiation.cc:769] Sending connection context
0621 07:51:31.006574 (+ 100us) client_negotiation.cc:240] Negotiation successful
0621 07:51:31.006694 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260621 07:51:31.006904 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:30.999744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41298 (local address 127.0.192.253:46609)
0621 07:51:30.999892 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:30.999896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:30.999915 (+ 19us) server_negotiation.cc:407] Connection header received
0621 07:51:30.999945 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:30.999947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:30.999980 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:31.000058 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:31.000397 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:31.001054 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:31.001715 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:31.001917 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:31.001986 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:31.003877 (+ 1891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:31.003880 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:31.003882 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:31.003901 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:31.005315 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:31.005719 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:31.005722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:31.005724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:31.005804 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:31.006066 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:31.006068 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:31.006069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:31.006284 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:31.006392 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:31.006642 (+ 250us) server_negotiation.cc:299] Negotiation successful
0621 07:51:31.006748 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260621 07:51:31.007598 23240 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42441 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:51:31.009011 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:31.000773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56780 (local address 127.0.192.254:37879)
0621 07:51:31.000859 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:31.000862 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:31.000888 (+ 26us) server_negotiation.cc:407] Connection header received
0621 07:51:31.000972 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:31.000974 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:31.001010 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:31.001060 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:31.001557 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:31.002098 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:31.002746 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:31.003047 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:31.003125 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:31.005731 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:31.005734 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:31.005736 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:31.005758 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:31.007336 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:31.007794 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:31.007799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:31.007800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:31.007892 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:31.008199 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:31.008203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:31.008205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:31.008379 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:31.008638 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:31.008761 (+ 123us) server_negotiation.cc:299] Negotiation successful
0621 07:51:31.008869 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
Jun 21 07:51:31 dist-test-slave-lzx3 krb5kdc[23009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028290, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.252@KRBTEST.COM for kudu/127.0.192.252@KRBTEST.COM
Jun 21 07:51:31 dist-test-slave-lzx3 krb5kdc[23009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.192.253@KRBTEST.COM
I20260621 07:51:31.015532 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:31.007971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:42441 (local address 127.0.0.1:43660)
0621 07:51:31.008155 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:31.008168 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:31.008241 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:31.008445 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:31.008447 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:31.008636 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:31.008700 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:31.008704 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:31.009555 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:31.009558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:31.010252 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:31.010264 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:31.010823 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:31.010826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:31.010882 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:31.011210 (+ 328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:31.011222 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:31.012461 (+ 1239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:31.014253 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:31.014255 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:31.014257 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:31.014431 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:31.014698 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:31.014704 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:31.014706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:31.014778 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:31.015142 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:31.015146 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:31.015258 (+ 112us) client_negotiation.cc:769] Sending connection context
0621 07:51:31.015357 (+ 99us) client_negotiation.cc:240] Negotiation successful
0621 07:51:31.015439 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260621 07:51:31.015750 23243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:31.008072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43660 (local address 127.0.192.252:42441)
0621 07:51:31.008152 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:31.008156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:31.008208 (+ 52us) server_negotiation.cc:407] Connection header received
0621 07:51:31.008306 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:31.008310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:31.008341 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:31.008436 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:31.008798 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:31.009440 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:31.010350 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:31.010720 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:31.010837 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:31.012642 (+ 1805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:31.012646 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:31.012648 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:31.012675 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:31.014142 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:31.014528 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:31.014533 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:31.014534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:31.014607 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:31.014903 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:31.014905 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:31.014906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:31.015044 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:31.015112 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:31.015433 (+ 321us) server_negotiation.cc:299] Negotiation successful
0621 07:51:31.015546 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260621 07:51:31.018452 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:31.010044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41302 (local address 127.0.192.253:46609)
0621 07:51:31.010138 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:31.010142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:31.010190 (+ 48us) server_negotiation.cc:407] Connection header received
0621 07:51:31.010284 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:31.010289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:31.010335 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:31.010405 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:31.011039 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:31.011693 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:31.012648 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:31.012831 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:31.012941 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:31.015278 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:31.015281 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:31.015282 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:31.015300 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:31.016934 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:31.017331 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:31.017336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:31.017341 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:31.017417 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:31.017694 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:31.017698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:31.017699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:31.017863 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:31.018073 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:31.018196 (+ 123us) server_negotiation.cc:299] Negotiation successful
0621 07:51:31.018308 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260621 07:51:31.018746 23240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 094f0fbfe0d84f018aba47c11847b821: Bootstrap starting.
I20260621 07:51:31.021219 23240 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 094f0fbfe0d84f018aba47c11847b821: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:31.021535 771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:51:31.021963 23240 log.cc:824] T 00000000000000000000000000000000 P 094f0fbfe0d84f018aba47c11847b821: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:31.022678 23064 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ec761274c9224942a6f2a02bb9cff3f7"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56780
I20260621 07:51:31.023490 23240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 094f0fbfe0d84f018aba47c11847b821: No bootstrap required, opened a new log
I20260621 07:51:31.026124 23240 raft_consensus.cc:348] T 00000000000000000000000000000000 P 094f0fbfe0d84f018aba47c11847b821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec761274c9224942a6f2a02bb9cff3f7" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 37879 } } peers { permanent_uuid: "f94652df8a564b99b52490162c57aa9e" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 46609 } } peers { permanent_uuid: "094f0fbfe0d84f018aba47c11847b821" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42441 } }
I20260621 07:51:31.026304 23240 raft_consensus.cc:729] T 00000000000000000000000000000000 P 094f0fbfe0d84f018aba47c11847b821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 094f0fbfe0d84f018aba47c11847b821, State: Initialized, Role: FOLLOWER
I20260621 07:51:31.026626 23240 consensus_queue.cc:260] T 00000000000000000000000000000000 P 094f0fbfe0d84f018aba47c11847b821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec761274c9224942a6f2a02bb9cff3f7" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 37879 } } peers { permanent_uuid: "f94652df8a564b99b52490162c57aa9e" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 46609 } } peers { permanent_uuid: "094f0fbfe0d84f018aba47c11847b821" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42441 } }
I20260621 07:51:31.027123 23246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 094f0fbfe0d84f018aba47c11847b821 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec761274c9224942a6f2a02bb9cff3f7" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 37879 } } peers { permanent_uuid: "f94652df8a564b99b52490162c57aa9e" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 46609 } } peers { permanent_uuid: "094f0fbfe0d84f018aba47c11847b821" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42441 } } }
I20260621 07:51:31.027261 23246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 094f0fbfe0d84f018aba47c11847b821 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:51:31.027758 23240 sys_catalog.cc:564] T 00000000000000000000000000000000 P 094f0fbfe0d84f018aba47c11847b821 [sys.catalog]: configured and running, proceeding with master startup.
W20260621 07:51:31.032224 23257 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 094f0fbfe0d84f018aba47c11847b821: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:51:31.032282 23257 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:51:31.030932 23172 master_runner.cc:186] Error getting master registration for 127.0.192.254:37879: OK, instance_id { permanent_uuid: "ec761274c9224942a6f2a02bb9cff3f7" instance_seqno: 1782028290605080 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 07:51:31.033677 23165 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42441 } attempt: 3
I20260621 07:51:31.033808 23172 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Jun 21 07:51:31 dist-test-slave-lzx3 krb5kdc[23009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028290, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.253@KRBTEST.COM for kudu/127.0.192.252@KRBTEST.COM
I20260621 07:51:31.042148 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:31.034000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:42441 (local address 127.0.0.1:43666)
0621 07:51:31.034135 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:31.034150 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:31.034241 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:31.034496 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:31.034499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:31.034682 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:31.034776 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:31.034782 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:31.035704 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:31.035710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:31.036422 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:31.036432 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:31.037025 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:31.037028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:31.037110 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:31.037612 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:31.037627 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:31.038989 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:31.040804 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:31.040806 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:31.040808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:31.041017 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:31.041274 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:31.041278 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:31.041281 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:31.041363 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:31.041743 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:31.041745 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:31.041821 (+ 76us) client_negotiation.cc:769] Sending connection context
0621 07:51:31.041908 (+ 87us) client_negotiation.cc:240] Negotiation successful
0621 07:51:31.042010 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260621 07:51:31.042204 23243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:31.034048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43666 (local address 127.0.192.252:42441)
0621 07:51:31.034160 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:31.034164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:31.034198 (+ 34us) server_negotiation.cc:407] Connection header received
0621 07:51:31.034339 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:31.034341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:31.034380 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:31.034460 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:31.034891 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:31.035604 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:31.036548 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:31.036900 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:31.037001 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:31.039130 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:31.039133 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:31.039135 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:31.039153 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:31.040708 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:31.041109 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:31.041114 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:31.041115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:31.041192 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:31.041445 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:31.041448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:31.041450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:31.041655 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:31.041746 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:31.041987 (+ 241us) server_negotiation.cc:299] Negotiation successful
0621 07:51:31.042082 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260621 07:51:31.044824 23165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f94652df8a564b99b52490162c57aa9e: Bootstrap starting.
I20260621 07:51:31.046983 23165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f94652df8a564b99b52490162c57aa9e: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:31.047716 23165 log.cc:824] T 00000000000000000000000000000000 P f94652df8a564b99b52490162c57aa9e: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:31.049362 23165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f94652df8a564b99b52490162c57aa9e: No bootstrap required, opened a new log
I20260621 07:51:31.051743 23165 raft_consensus.cc:348] T 00000000000000000000000000000000 P f94652df8a564b99b52490162c57aa9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec761274c9224942a6f2a02bb9cff3f7" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 37879 } } peers { permanent_uuid: "f94652df8a564b99b52490162c57aa9e" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 46609 } } peers { permanent_uuid: "094f0fbfe0d84f018aba47c11847b821" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42441 } }
I20260621 07:51:31.051923 23165 raft_consensus.cc:729] T 00000000000000000000000000000000 P f94652df8a564b99b52490162c57aa9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f94652df8a564b99b52490162c57aa9e, State: Initialized, Role: FOLLOWER
I20260621 07:51:31.052168 23165 consensus_queue.cc:260] T 00000000000000000000000000000000 P f94652df8a564b99b52490162c57aa9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec761274c9224942a6f2a02bb9cff3f7" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 37879 } } peers { permanent_uuid: "f94652df8a564b99b52490162c57aa9e" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 46609 } } peers { permanent_uuid: "094f0fbfe0d84f018aba47c11847b821" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42441 } }
I20260621 07:51:31.052524 23260 sys_catalog.cc:455] T 00000000000000000000000000000000 P f94652df8a564b99b52490162c57aa9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec761274c9224942a6f2a02bb9cff3f7" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 37879 } } peers { permanent_uuid: "f94652df8a564b99b52490162c57aa9e" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 46609 } } peers { permanent_uuid: "094f0fbfe0d84f018aba47c11847b821" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42441 } } }
I20260621 07:51:31.052659 23260 sys_catalog.cc:458] T 00000000000000000000000000000000 P f94652df8a564b99b52490162c57aa9e [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:51:31.053157 23165 sys_catalog.cc:564] T 00000000000000000000000000000000 P f94652df8a564b99b52490162c57aa9e [sys.catalog]: configured and running, proceeding with master startup.
W20260621 07:51:31.057431 23271 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f94652df8a564b99b52490162c57aa9e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:51:31.057504 23271 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:51:31.056129 23097 master_runner.cc:186] Error getting master registration for 127.0.192.254:37879: OK, instance_id { permanent_uuid: "ec761274c9224942a6f2a02bb9cff3f7" instance_seqno: 1782028290605080 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 07:51:31.059106 23097 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 07:51:31.061466 23090 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42441 } attempt: 3
Jun 21 07:51:31 dist-test-slave-lzx3 krb5kdc[23009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028290, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for kudu/127.0.192.252@KRBTEST.COM
I20260621 07:51:31.069376 23243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:31.061902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43674 (local address 127.0.192.252:42441)
0621 07:51:31.061996 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:31.062000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:31.062036 (+ 36us) server_negotiation.cc:407] Connection header received
0621 07:51:31.062161 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:31.062164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:31.062198 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:31.062281 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:31.062680 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:31.063393 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:31.064102 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:31.064436 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:31.064515 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:31.066297 (+ 1782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:31.066300 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:31.066303 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:31.066323 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:31.067820 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:31.068232 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:31.068237 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:31.068239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:31.068317 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:31.068643 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:31.068645 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:31.068647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:31.068837 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:31.068916 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:31.069157 (+ 241us) server_negotiation.cc:299] Negotiation successful
0621 07:51:31.069251 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260621 07:51:31.069375 23094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:31.061827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:42441 (local address 127.0.0.1:43674)
0621 07:51:31.061981 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:31.061993 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:31.062088 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:31.062294 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:31.062297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:31.062484 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:31.062574 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:31.062579 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:31.063481 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:31.063484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:31.063985 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:31.063992 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:31.064523 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:31.064527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:31.064626 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:31.065021 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:31.065033 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:31.066173 (+ 1140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:31.067918 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:31.067921 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:31.067923 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:31.068130 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:31.068403 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:31.068419 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:31.068421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:31.068536 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:31.068929 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:31.068933 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:31.069012 (+ 79us) client_negotiation.cc:769] Sending connection context
0621 07:51:31.069111 (+ 99us) client_negotiation.cc:240] Negotiation successful
0621 07:51:31.069227 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260621 07:51:31.072391 23090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec761274c9224942a6f2a02bb9cff3f7: Bootstrap starting.
I20260621 07:51:31.074780 23090 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec761274c9224942a6f2a02bb9cff3f7: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:31.075567 23090 log.cc:824] T 00000000000000000000000000000000 P ec761274c9224942a6f2a02bb9cff3f7: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:31.077427 23090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec761274c9224942a6f2a02bb9cff3f7: No bootstrap required, opened a new log
I20260621 07:51:31.080098 23090 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec761274c9224942a6f2a02bb9cff3f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec761274c9224942a6f2a02bb9cff3f7" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 37879 } } peers { permanent_uuid: "f94652df8a564b99b52490162c57aa9e" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 46609 } } peers { permanent_uuid: "094f0fbfe0d84f018aba47c11847b821" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42441 } }
I20260621 07:51:31.080279 23090 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec761274c9224942a6f2a02bb9cff3f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec761274c9224942a6f2a02bb9cff3f7, State: Initialized, Role: FOLLOWER
I20260621 07:51:31.080547 23090 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec761274c9224942a6f2a02bb9cff3f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec761274c9224942a6f2a02bb9cff3f7" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 37879 } } peers { permanent_uuid: "f94652df8a564b99b52490162c57aa9e" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 46609 } } peers { permanent_uuid: "094f0fbfe0d84f018aba47c11847b821" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42441 } }
I20260621 07:51:31.081630 23090 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec761274c9224942a6f2a02bb9cff3f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:51:31.081700 23064 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec761274c9224942a6f2a02bb9cff3f7 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260621 07:51:31.081774 23064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec761274c9224942a6f2a02bb9cff3f7 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:31.082818 23064 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec761274c9224942a6f2a02bb9cff3f7 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec761274c9224942a6f2a02bb9cff3f7" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 37879 } } peers { permanent_uuid: "f94652df8a564b99b52490162c57aa9e" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 46609 } } peers { permanent_uuid: "094f0fbfe0d84f018aba47c11847b821" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42441 } }
I20260621 07:51:31.083681 23064 leader_election.cc:288] T 00000000000000000000000000000000 P ec761274c9224942a6f2a02bb9cff3f7 [CANDIDATE]: Term 1 election: Requested vote from peers f94652df8a564b99b52490162c57aa9e (127.0.192.253:46609), 094f0fbfe0d84f018aba47c11847b821 (127.0.192.252:42441)
I20260621 07:51:31.084435 23214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec761274c9224942a6f2a02bb9cff3f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "094f0fbfe0d84f018aba47c11847b821"
I20260621 07:51:31.084679 23214 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 094f0fbfe0d84f018aba47c11847b821 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:31.084758 23139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec761274c9224942a6f2a02bb9cff3f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f94652df8a564b99b52490162c57aa9e"
I20260621 07:51:31.084936 23139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f94652df8a564b99b52490162c57aa9e [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:31.085894 23214 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 094f0fbfe0d84f018aba47c11847b821 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec761274c9224942a6f2a02bb9cff3f7 in term 1.
I20260621 07:51:31.086108 23139 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f94652df8a564b99b52490162c57aa9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec761274c9224942a6f2a02bb9cff3f7 in term 1.
I20260621 07:51:31.086266 23040 leader_election.cc:302] T 00000000000000000000000000000000 P ec761274c9224942a6f2a02bb9cff3f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 094f0fbfe0d84f018aba47c11847b821, ec761274c9224942a6f2a02bb9cff3f7; no voters:
I20260621 07:51:31.087779 23274 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec761274c9224942a6f2a02bb9cff3f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec761274c9224942a6f2a02bb9cff3f7" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 37879 } } peers { permanent_uuid: "f94652df8a564b99b52490162c57aa9e" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 46609 } } peers { permanent_uuid: "094f0fbfe0d84f018aba47c11847b821" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42441 } } }
I20260621 07:51:31.087895 23274 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec761274c9224942a6f2a02bb9cff3f7 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:51:31.087981 23274 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec761274c9224942a6f2a02bb9cff3f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:31.088279 23274 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec761274c9224942a6f2a02bb9cff3f7 [term 1 LEADER]: Becoming Leader. State: Replica: ec761274c9224942a6f2a02bb9cff3f7, State: Running, Role: LEADER
I20260621 07:51:31.088557 23274 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec761274c9224942a6f2a02bb9cff3f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec761274c9224942a6f2a02bb9cff3f7" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 37879 } } peers { permanent_uuid: "f94652df8a564b99b52490162c57aa9e" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 46609 } } peers { permanent_uuid: "094f0fbfe0d84f018aba47c11847b821" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42441 } }
I20260621 07:51:31.090123 23274 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec761274c9224942a6f2a02bb9cff3f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec761274c9224942a6f2a02bb9cff3f7. Latest consensus state: current_term: 1 leader_uuid: "ec761274c9224942a6f2a02bb9cff3f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec761274c9224942a6f2a02bb9cff3f7" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 37879 } } peers { permanent_uuid: "f94652df8a564b99b52490162c57aa9e" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 46609 } } peers { permanent_uuid: "094f0fbfe0d84f018aba47c11847b821" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42441 } } }
I20260621 07:51:31.090217 23274 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec761274c9224942a6f2a02bb9cff3f7 [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:31.091717 23022 master_runner.cc:263] No leader master found from master ec761274c9224942a6f2a02bb9cff3f7
I20260621 07:51:31.091773 23022 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260621 07:51:31.093142 23285 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ec761274c9224942a6f2a02bb9cff3f7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:51:31.093248 23285 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:51:31.101619 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:31.088867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56790 (local address 127.0.192.254:37879)
0621 07:51:31.088962 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:31.088965 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:31.093459 (+ 4494us) server_negotiation.cc:407] Connection header received
0621 07:51:31.093543 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:31.093545 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:31.093583 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:31.093649 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:31.094603 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:31.095367 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:31.096399 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:31.096710 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:31.096774 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:31.098537 (+ 1763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:31.098540 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:31.098542 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:31.098563 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:31.099987 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:31.100422 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:31.100426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:31.100427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:31.100502 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:31.100797 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:31.100800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:31.100802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:31.100980 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:31.101076 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:31.101393 (+ 317us) server_negotiation.cc:299] Negotiation successful
0621 07:51:31.101488 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260621 07:51:31.102779 23043 tablet.cc:2369] T 00000000000000000000000000000000 P ec761274c9224942a6f2a02bb9cff3f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:51:31.109375 23243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:31.090005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43676 (local address 127.0.192.252:42441)
0621 07:51:31.090089 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:31.090093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:31.100811 (+ 10718us) server_negotiation.cc:407] Connection header received
0621 07:51:31.100972 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:31.100975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:31.101015 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:31.101061 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:31.101834 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:31.102594 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:31.104123 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:31.104470 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:31.104604 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:31.106127 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:31.106130 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:31.106132 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:31.106156 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:31.107603 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:31.108045 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:31.108049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:31.108051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:31.108125 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:31.108413 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:31.108417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:31.108419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:31.108613 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:31.108709 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:31.109155 (+ 446us) server_negotiation.cc:299] Negotiation successful
0621 07:51:31.109242 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260621 07:51:31.109773 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:31.089384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41308 (local address 127.0.192.253:46609)
0621 07:51:31.089475 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:31.089478 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:31.093263 (+ 3785us) server_negotiation.cc:407] Connection header received
0621 07:51:31.096730 (+ 3467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:31.096733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:31.096771 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:31.096813 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:31.097675 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:31.098416 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:31.099405 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:31.099689 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:31.099745 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:31.101146 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:31.101149 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:31.101151 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:31.101173 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:31.102698 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:31.108653 (+ 5955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:31.108656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:31.108658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:31.108735 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:31.109001 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:31.109004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:31.109006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:31.109174 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:31.109262 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:31.109554 (+ 292us) server_negotiation.cc:299] Negotiation successful
0621 07:51:31.109649 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260621 07:51:31.112115 23295 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:51:31.112881 23295 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:51:31.119673 23214 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 094f0fbfe0d84f018aba47c11847b821 [term 1 FOLLOWER]: Refusing update from remote peer ec761274c9224942a6f2a02bb9cff3f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:51:31.119715 23139 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f94652df8a564b99b52490162c57aa9e [term 1 FOLLOWER]: Refusing update from remote peer ec761274c9224942a6f2a02bb9cff3f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:51:31.120266 23274 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ec761274c9224942a6f2a02bb9cff3f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "094f0fbfe0d84f018aba47c11847b821" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:51:31.120654 23286 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ec761274c9224942a6f2a02bb9cff3f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f94652df8a564b99b52490162c57aa9e" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 46609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:51:31.123311 23246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 094f0fbfe0d84f018aba47c11847b821 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec761274c9224942a6f2a02bb9cff3f7. Latest consensus state: current_term: 1 leader_uuid: "ec761274c9224942a6f2a02bb9cff3f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec761274c9224942a6f2a02bb9cff3f7" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 37879 } } peers { permanent_uuid: "f94652df8a564b99b52490162c57aa9e" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 46609 } } peers { permanent_uuid: "094f0fbfe0d84f018aba47c11847b821" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42441 } } }
I20260621 07:51:31.123406 23246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 094f0fbfe0d84f018aba47c11847b821 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:51:31.123508 23260 sys_catalog.cc:455] T 00000000000000000000000000000000 P f94652df8a564b99b52490162c57aa9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec761274c9224942a6f2a02bb9cff3f7. Latest consensus state: current_term: 1 leader_uuid: "ec761274c9224942a6f2a02bb9cff3f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec761274c9224942a6f2a02bb9cff3f7" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 37879 } } peers { permanent_uuid: "f94652df8a564b99b52490162c57aa9e" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 46609 } } peers { permanent_uuid: "094f0fbfe0d84f018aba47c11847b821" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42441 } } }
I20260621 07:51:31.123605 23260 sys_catalog.cc:458] T 00000000000000000000000000000000 P f94652df8a564b99b52490162c57aa9e [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:51:31.131968 23274 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec761274c9224942a6f2a02bb9cff3f7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ec761274c9224942a6f2a02bb9cff3f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec761274c9224942a6f2a02bb9cff3f7" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 37879 } } peers { permanent_uuid: "f94652df8a564b99b52490162c57aa9e" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 46609 } } peers { permanent_uuid: "094f0fbfe0d84f018aba47c11847b821" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42441 } } }
I20260621 07:51:31.132057 23274 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec761274c9224942a6f2a02bb9cff3f7 [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:31.132658 23260 sys_catalog.cc:455] T 00000000000000000000000000000000 P f94652df8a564b99b52490162c57aa9e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ec761274c9224942a6f2a02bb9cff3f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec761274c9224942a6f2a02bb9cff3f7" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 37879 } } peers { permanent_uuid: "f94652df8a564b99b52490162c57aa9e" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 46609 } } peers { permanent_uuid: "094f0fbfe0d84f018aba47c11847b821" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42441 } } }
I20260621 07:51:31.132772 23260 sys_catalog.cc:458] T 00000000000000000000000000000000 P f94652df8a564b99b52490162c57aa9e [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:51:31.132901 23297 mvcc.cc:205] Tried to move back new op lower bound from 7299187880419491840 to 7299187880302100480. Current Snapshot: MvccSnapshot[applied={T|T < 7299187880419491840}]
I20260621 07:51:31.133029 23299 mvcc.cc:205] Tried to move back new op lower bound from 7299187880419491840 to 7299187880302100480. Current Snapshot: MvccSnapshot[applied={T|T < 7299187880419491840}]
I20260621 07:51:31.133776 23295 catalog_manager.cc:1347] Generated new cluster ID: cde9ff03ea804260be4827fce0583697
I20260621 07:51:31.133836 23295 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:51:31.140990 23286 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec761274c9224942a6f2a02bb9cff3f7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ec761274c9224942a6f2a02bb9cff3f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec761274c9224942a6f2a02bb9cff3f7" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 37879 } } peers { permanent_uuid: "f94652df8a564b99b52490162c57aa9e" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 46609 } } peers { permanent_uuid: "094f0fbfe0d84f018aba47c11847b821" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42441 } } }
I20260621 07:51:31.141083 23286 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec761274c9224942a6f2a02bb9cff3f7 [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:31.141920 23246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 094f0fbfe0d84f018aba47c11847b821 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ec761274c9224942a6f2a02bb9cff3f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec761274c9224942a6f2a02bb9cff3f7" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 37879 } } peers { permanent_uuid: "f94652df8a564b99b52490162c57aa9e" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 46609 } } peers { permanent_uuid: "094f0fbfe0d84f018aba47c11847b821" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 42441 } } }
I20260621 07:51:31.142010 23246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 094f0fbfe0d84f018aba47c11847b821 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:51:31.143004 23298 mvcc.cc:205] Tried to move back new op lower bound from 7299187880419491840 to 7299187880302100480. Current Snapshot: MvccSnapshot[applied={T|T < 7299187880419491840}]
I20260621 07:51:31.168457 23295 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:51:31.169584 23295 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:51:31.185053 23295 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec761274c9224942a6f2a02bb9cff3f7: Generated new TSK 0
I20260621 07:51:31.185793 23295 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 07:51:31.220247 23170 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:51:31.217359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41324 (local address 127.0.192.253:46609)
0621 07:51:31.217485 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:31.217489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:31.219253 (+ 1764us) server_negotiation.cc:407] Connection header received
0621 07:51:31.219436 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:31.219439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:31.219486 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:31.219536 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:31.220125 (+ 589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41324: BlockingRecv error: recv got EOF from 127.0.0.1:41324 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260621 07:51:31.221725 23093 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:51:31.218108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56804 (local address 127.0.192.254:37879)
0621 07:51:31.218217 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:31.218221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:31.220602 (+ 2381us) server_negotiation.cc:407] Connection header received
0621 07:51:31.221055 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:31.221058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:31.221102 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:31.221147 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:31.221621 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56804: BlockingRecv error: recv got EOF from 127.0.0.1:56804 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260621 07:51:31.222800 23243 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:51:31.218687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43690 (local address 127.0.192.252:42441)
0621 07:51:31.218800 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:31.218804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:31.220986 (+ 2182us) server_negotiation.cc:407] Connection header received
0621 07:51:31.222121 (+ 1135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:31.222125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:31.222171 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:31.222218 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:31.222661 (+ 443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43690: BlockingRecv error: recv got EOF from 127.0.0.1:43690 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260621 07:51:31.226925 23170 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:51:31.225638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41338 (local address 127.0.192.253:46609)
0621 07:51:31.225726 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:31.225730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:31.226113 (+ 383us) server_negotiation.cc:407] Connection header received
0621 07:51:31.226296 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:31.226299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:31.226361 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:31.226407 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:31.226842 (+ 435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41338: BlockingRecv error: recv got EOF from 127.0.0.1:41338 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260621 07:51:31.231595 23170 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:51:31.230197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41346 (local address 127.0.192.253:46609)
0621 07:51:31.230324 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:31.230328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:31.230800 (+ 472us) server_negotiation.cc:407] Connection header received
0621 07:51:31.230962 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:31.230965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:31.231007 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:31.231051 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:31.231487 (+ 436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41346: BlockingRecv error: recv got EOF from 127.0.0.1:41346 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260621 07:51:31.236924 23170 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:51:31.235457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41354 (local address 127.0.192.253:46609)
0621 07:51:31.235545 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:31.235549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:31.236069 (+ 520us) server_negotiation.cc:407] Connection header received
0621 07:51:31.236244 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:31.236247 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:31.236291 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:31.236351 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:31.236843 (+ 492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41354: BlockingRecv error: recv got EOF from 127.0.0.1:41354 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260621 07:51:31.243799 23170 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:51:31.242871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41370 (local address 127.0.192.253:46609)
0621 07:51:31.242985 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:31.242988 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:31.243188 (+ 200us) server_negotiation.cc:407] Connection header received
0621 07:51:31.243259 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:31.243261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:31.243301 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:31.243374 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:31.243716 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41370: BlockingRecv error: recv got EOF from 127.0.0.1:41370 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260621 07:51:31.254834 23170 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:51:31.253666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41384 (local address 127.0.192.253:46609)
0621 07:51:31.253779 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:31.253783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:31.253965 (+ 182us) server_negotiation.cc:407] Connection header received
0621 07:51:31.254138 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:31.254141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:31.254193 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:31.254286 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:31.254736 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41384: BlockingRecv error: recv got EOF from 127.0.0.1:41384 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260621 07:51:31.273497 23170 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:51:31.272636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41390 (local address 127.0.192.253:46609)
0621 07:51:31.272722 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:31.272726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:31.272878 (+ 152us) server_negotiation.cc:407] Connection header received
0621 07:51:31.272951 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:31.272953 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:31.272994 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:31.273071 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:31.273413 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41390: BlockingRecv error: recv got EOF from 127.0.0.1:41390 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260621 07:51:31.308007 23170 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:51:31.307053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41400 (local address 127.0.192.253:46609)
0621 07:51:31.307193 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:31.307196 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:31.307274 (+ 78us) server_negotiation.cc:407] Connection header received
0621 07:51:31.307381 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:31.307383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:31.307423 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:31.307491 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:31.307884 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41400: BlockingRecv error: recv got EOF from 127.0.0.1:41400 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260621 07:51:31.375022 23170 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:51:31.373768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41406 (local address 127.0.192.253:46609)
0621 07:51:31.373892 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:31.373896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:31.374088 (+ 192us) server_negotiation.cc:407] Connection header received
0621 07:51:31.374172 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:31.374175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:31.374219 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:31.374515 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:31.374915 (+ 400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41406: BlockingRecv error: recv got EOF from 127.0.0.1:41406 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260621 07:51:31.506872 23170 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:51:31.505561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41422 (local address 127.0.192.253:46609)
0621 07:51:31.505738 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:31.505745 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:31.505898 (+ 153us) server_negotiation.cc:407] Connection header received
0621 07:51:31.506048 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:31.506053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:31.506125 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:31.506244 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:31.506718 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41422: BlockingRecv error: recv got EOF from 127.0.0.1:41422 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260621 07:51:31.766276 23170 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:51:31.765293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41430 (local address 127.0.192.253:46609)
0621 07:51:31.765427 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:31.765430 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:31.765538 (+ 108us) server_negotiation.cc:407] Connection header received
0621 07:51:31.765663 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:31.765665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:31.765710 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:31.765766 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:31.766148 (+ 382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41430: BlockingRecv error: recv got EOF from 127.0.0.1:41430 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 07:51:32.033280 23257 catalog_manager.cc:1259] Loaded cluster ID: cde9ff03ea804260be4827fce0583697
I20260621 07:51:32.033370 23257 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 094f0fbfe0d84f018aba47c11847b821: loading cluster ID for follower catalog manager: success
I20260621 07:51:32.034929 23257 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 094f0fbfe0d84f018aba47c11847b821: acquiring CA information for follower catalog manager: success
I20260621 07:51:32.035976 23257 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 094f0fbfe0d84f018aba47c11847b821: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 07:51:32.058499 23271 catalog_manager.cc:1259] Loaded cluster ID: cde9ff03ea804260be4827fce0583697
I20260621 07:51:32.058554 23271 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f94652df8a564b99b52490162c57aa9e: loading cluster ID for follower catalog manager: success
I20260621 07:51:32.059857 23271 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f94652df8a564b99b52490162c57aa9e: acquiring CA information for follower catalog manager: success
I20260621 07:51:32.060864 23271 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f94652df8a564b99b52490162c57aa9e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 07:51:32.284013 23371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:32.280213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41436 (local address 127.0.192.253:46609)
0621 07:51:32.280461 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:32.280465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:32.280491 (+ 26us) server_negotiation.cc:407] Connection header received
0621 07:51:32.280646 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:32.280649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:32.280771 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:32.280879 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:51:32.281156 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:32.281643 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:32.282567 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:32.282812 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:32.282906 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:32.283215 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:51:32.283633 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:51:32.283729 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:32.283796 (+ 67us) server_negotiation.cc:299] Negotiation successful
0621 07:51:32.283838 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":79,"threads_started":1}
I20260621 07:51:32.290017 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:32.286640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43706 (local address 127.0.192.252:42441)
0621 07:51:32.286865 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:32.286868 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:32.286880 (+ 12us) server_negotiation.cc:407] Connection header received
0621 07:51:32.286989 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:32.286991 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:32.287142 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:32.287236 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:51:32.287406 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:32.287916 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:32.288754 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:32.288976 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:32.289052 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:32.289276 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:51:32.289650 (+ 374us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:51:32.289756 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:32.289803 (+ 47us) server_negotiation.cc:299] Negotiation successful
0621 07:51:32.289842 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":63,"threads_started":1}
I20260621 07:51:32.296532 23371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:32.292646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41448 (local address 127.0.192.253:46609)
0621 07:51:32.292742 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:32.292746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:32.293070 (+ 324us) server_negotiation.cc:407] Connection header received
0621 07:51:32.293231 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:32.293234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:32.293273 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:32.293312 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:51:32.293617 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:32.294330 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:32.295442 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:32.295729 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:32.295816 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:32.296075 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:51:32.296137 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:51:32.296205 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:32.296351 (+ 146us) server_negotiation.cc:299] Negotiation successful
0621 07:51:32.296397 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260621 07:51:32.296897 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:32.292837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56816 (local address 127.0.192.254:37879)
0621 07:51:32.293095 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:32.293101 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:32.293117 (+ 16us) server_negotiation.cc:407] Connection header received
0621 07:51:32.293278 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:32.293281 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:32.293430 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:32.293544 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:51:32.293841 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:32.294534 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:32.295565 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:32.295853 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:32.295971 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:32.296231 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:51:32.296481 (+ 250us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:51:32.296588 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:32.296670 (+ 82us) server_negotiation.cc:299] Negotiation successful
0621 07:51:32.296733 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":67,"threads_started":1}
I20260621 07:51:32.297158 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:32.293145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43720 (local address 127.0.192.252:42441)
0621 07:51:32.293240 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:32.293243 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:32.293552 (+ 309us) server_negotiation.cc:407] Connection header received
0621 07:51:32.293746 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:32.293749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:32.293793 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:32.293862 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:51:32.294173 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:32.294814 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:32.295655 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:32.295919 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:32.295993 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:32.296300 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:51:32.296380 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:51:32.296436 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:32.296964 (+ 528us) server_negotiation.cc:299] Negotiation successful
0621 07:51:32.297027 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260621 07:51:32.299877 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 23022
I20260621 07:51:32.305857 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 23097
I20260621 07:51:32.311118 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 23172
2026-06-21T07:51:32Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2911 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:51:32 dist-test-slave-lzx3 krb5kdc[23388](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:51:32 dist-test-slave-lzx3 krb5kdc[23388](info): set up 2 sockets
Jun 21 07:51:32 dist-test-slave-lzx3 krb5kdc[23388](info): commencing operation
krb5kdc: starting...
W20260621 07:51:33.268822 771 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/krb5kdc/test-user.keytab.
Jun 21 07:51:33 dist-test-slave-lzx3 krb5kdc[23388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:51:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:51:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.192.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
Entry for principal kudu/127.0.192.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab.
I20260621 07:51:33.374964 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:34805
--webserver_interface=127.0.192.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:41147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:34805,127.0.192.253:32909,127.0.192.252:44291
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:51:33.472043 23401 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 07:51:33.472275 23401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:33.472294 23401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:33.472384 23401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:33.476262 23401 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:51:33.476310 23401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:33.476341 23401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:33.476356 23401 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:51:33.476369 23401 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:51:33.481163 23401 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:41147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:34805,127.0.192.253:32909,127.0.192.252:44291
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/krb5kdc/kudu_127.0.192.254.keytab
--principal=kudu/127.0.192.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.254:34805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.192.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:33.481983 23401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:33.482858 23401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:33.490715 23409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:33.490836 23401 server_base.cc:1034] running on GCE node
W20260621 07:51:33.490716 23406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:33.490775 23407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:33.491401 23401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:33.492238 23401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:33.493413 23401 hybrid_clock.cc:648] HybridClock initialized: now 1782028293493401 us; error 29 us; skew 500 ppm
Jun 21 07:51:33 dist-test-slave-lzx3 krb5kdc[23388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:33.495802 23401 init.cc:377] Logged in from keytab as kudu/127.0.192.254@KRBTEST.COM (short username kudu)
I20260621 07:51:33.496532 23401 webserver.cc:460] Webserver started at http://127.0.192.254:44257/ using document root <none> and password file <none>
I20260621 07:51:33.497109 23401 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:33.497195 23401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:33.497399 23401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:33.499179 23401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-0/data/instance:
uuid: "371e06a939234fd59161dfbd663a8351"
format_stamp: "Formatted at 2026-06-21 07:51:33 on dist-test-slave-lzx3"
server_key: "d1690d135dc7fdfe6336f74561c81b8e"
server_key_iv: "dc74b2f009e9810fb078bc74f5b7aae2"
server_key_version: "encryptionkey@0"
I20260621 07:51:33.499668 23401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-0/wal/instance:
uuid: "371e06a939234fd59161dfbd663a8351"
format_stamp: "Formatted at 2026-06-21 07:51:33 on dist-test-slave-lzx3"
server_key: "d1690d135dc7fdfe6336f74561c81b8e"
server_key_iv: "dc74b2f009e9810fb078bc74f5b7aae2"
server_key_version: "encryptionkey@0"
I20260621 07:51:33.502627 23401 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 07:51:33.504665 23416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:33.505693 23401 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:51:33.505810 23401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-0/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-0/wal
uuid: "371e06a939234fd59161dfbd663a8351"
format_stamp: "Formatted at 2026-06-21 07:51:33 on dist-test-slave-lzx3"
server_key: "d1690d135dc7fdfe6336f74561c81b8e"
server_key_iv: "dc74b2f009e9810fb078bc74f5b7aae2"
server_key_version: "encryptionkey@0"
I20260621 07:51:33.505899 23401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:33.520947 23401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:33.523135 23401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:33.523255 23401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:33.529057 23401 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.254:34805
I20260621 07:51:33.529302 23468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.254:34805 every 8 connection(s)
I20260621 07:51:33.529850 23401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-0/data/info.pb
I20260621 07:51:33.532325 23469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:33.535498 23469 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34805 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:51:33.539525 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 23401
I20260621 07:51:33.539656 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-0/wal/instance
I20260621 07:51:33.539944 771 external_mini_cluster.cc:1428] Setting key fb43273977edd7d4491cdd6f4be231a4
Jun 21 07:51:33 dist-test-slave-lzx3 krb5kdc[23388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.192.253@KRBTEST.COM; defaulting to no policy
I20260621 07:51:33.549286 23471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:33.537404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:34805 (local address 127.0.0.1:35992)
0621 07:51:33.537850 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:33.537887 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:33.538575 (+ 688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:33.539143 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:33.539147 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:33.539513 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:33.539831 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:33.539847 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:33.541107 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:33.541112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:33.541819 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:33.541827 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:33.542504 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:33.542507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:33.542595 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:33.543123 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:33.543137 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:33.544409 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:33.546612 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:33.546620 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:33.546626 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:33.546859 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:33.547226 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:33.547228 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:33.547229 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:33.547330 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:33.548014 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:33.548019 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:33.548129 (+ 110us) client_negotiation.cc:769] Sending connection context
0621 07:51:33.548313 (+ 184us) client_negotiation.cc:240] Negotiation successful
0621 07:51:33.548534 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":136,"threads_started":1}
I20260621 07:51:33.549288 23472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:33.537404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35992 (local address 127.0.192.254:34805)
0621 07:51:33.538208 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:33.538215 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:33.538244 (+ 29us) server_negotiation.cc:407] Connection header received
0621 07:51:33.538802 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:33.538805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:33.538992 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:33.539121 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:33.539985 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:33.540986 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:33.541940 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:33.542378 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:33.542495 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:33.544576 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:33.544583 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:33.544597 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:33.544625 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:33.546481 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:33.546975 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:33.546983 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:33.546987 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:33.547122 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:33.547422 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:33.547426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:33.547427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:33.547896 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:33.548033 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:33.548405 (+ 372us) server_negotiation.cc:299] Negotiation successful
0621 07:51:33.548650 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":719,"spinlock_wait_cycles":67840,"thread_start_us":136,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.253@KRBTEST.COM" created.
I20260621 07:51:33.551380 23469 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.192.253" port: 32909 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:51:33.551980 23471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:33.551767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.253:32909 (local address 127.0.0.1:46274)
0621 07:51:33.551863 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:33.551932 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.253:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260621 07:51:33.552208 23419 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.192.253:32909: connect: Connection refused (error 111)
I20260621 07:51:33.553059 23471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:33.552880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.253:32909 (local address 127.0.0.1:46280)
0621 07:51:33.552974 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:33.553013 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.253:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260621 07:51:33.553321 23469 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.192.253:32909: Network error: Client connection negotiation failed: client connection to 127.0.192.253:32909: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/krb5kdc/kudu_127.0.192.253.keytab.
Entry for principal kudu/127.0.192.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/krb5kdc/kudu_127.0.192.253.keytab.
I20260621 07:51:33.565287 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.253:32909
--webserver_interface=127.0.192.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/krb5kdc/kudu_127.0.192.253.keytab
--principal=kudu/127.0.192.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:41147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:34805,127.0.192.253:32909,127.0.192.252:44291
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:51:33.602582 23469 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.192.253" port: 32909 } attempt: 1
I20260621 07:51:33.603367 23471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:33.603103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.253:32909 (local address 127.0.0.1:46284)
0621 07:51:33.603247 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:33.603299 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.253:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260621 07:51:33.604033 23471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:33.603860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.253:32909 (local address 127.0.0.1:46290)
0621 07:51:33.603948 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:33.603987 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.253:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260621 07:51:33.604212 23469 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.192.253:32909: Network error: Client connection negotiation failed: client connection to 127.0.192.253:32909: connect: Connection refused (error 111)
W20260621 07:51:33.662884 23476 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 07:51:33.663081 23476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:33.663100 23476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:33.663192 23476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:33.666770 23476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:51:33.666816 23476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:33.666836 23476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:33.666891 23476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:51:33.666927 23476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:51:33.671445 23476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:41147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:34805,127.0.192.253:32909,127.0.192.252:44291
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/krb5kdc/kudu_127.0.192.253.keytab
--principal=kudu/127.0.192.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.253:32909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.192.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:33.672401 23476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:33.672370 23469 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.192.253" port: 32909 } attempt: 2
I20260621 07:51:33.673198 23471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:33.672843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.253:32909 (local address 127.0.0.1:46304)
0621 07:51:33.673026 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:33.673105 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.253:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260621 07:51:33.673560 23476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 07:51:33.674189 23471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:33.673822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.253:32909 (local address 127.0.0.1:46306)
0621 07:51:33.673981 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:33.674082 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.253:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260621 07:51:33.674409 23469 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.192.253:32909: Network error: Client connection negotiation failed: client connection to 127.0.192.253:32909: connect: Connection refused (error 111)
W20260621 07:51:33.682164 23484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:33.682271 23481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:33.682374 23476 server_base.cc:1034] running on GCE node
W20260621 07:51:33.682178 23482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:33.682883 23476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:33.683720 23476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:33.684885 23476 hybrid_clock.cc:648] HybridClock initialized: now 1782028293684880 us; error 24 us; skew 500 ppm
Jun 21 07:51:33 dist-test-slave-lzx3 krb5kdc[23388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:33.687382 23476 init.cc:377] Logged in from keytab as kudu/127.0.192.253@KRBTEST.COM (short username kudu)
I20260621 07:51:33.688154 23476 webserver.cc:460] Webserver started at http://127.0.192.253:45249/ using document root <none> and password file <none>
I20260621 07:51:33.688632 23476 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:33.688689 23476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:33.688828 23476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:33.690475 23476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-1/data/instance:
uuid: "942def5184454e568c77a7b32b42cffc"
format_stamp: "Formatted at 2026-06-21 07:51:33 on dist-test-slave-lzx3"
server_key: "ce61958797340b8d3adef42011fa74da"
server_key_iv: "2ad0c68bb8b994d7ed1ae086ae05ccd5"
server_key_version: "encryptionkey@0"
I20260621 07:51:33.690927 23476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-1/wal/instance:
uuid: "942def5184454e568c77a7b32b42cffc"
format_stamp: "Formatted at 2026-06-21 07:51:33 on dist-test-slave-lzx3"
server_key: "ce61958797340b8d3adef42011fa74da"
server_key_iv: "2ad0c68bb8b994d7ed1ae086ae05ccd5"
server_key_version: "encryptionkey@0"
I20260621 07:51:33.694247 23476 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 07:51:33.696365 23491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:33.697510 23476 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:51:33.697609 23476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-1/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-1/wal
uuid: "942def5184454e568c77a7b32b42cffc"
format_stamp: "Formatted at 2026-06-21 07:51:33 on dist-test-slave-lzx3"
server_key: "ce61958797340b8d3adef42011fa74da"
server_key_iv: "2ad0c68bb8b994d7ed1ae086ae05ccd5"
server_key_version: "encryptionkey@0"
I20260621 07:51:33.697685 23476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:33.721629 23476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:33.724244 23476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:33.724367 23476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:33.730190 23476 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.253:32909
I20260621 07:51:33.730453 23543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.253:32909 every 8 connection(s)
I20260621 07:51:33.731089 23476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-1/data/info.pb
I20260621 07:51:33.733350 23544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:33.736438 23544 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34805 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:51:33.740594 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 23476
I20260621 07:51:33.740713 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-1/wal/instance
I20260621 07:51:33.740985 771 external_mini_cluster.cc:1428] Setting key e44bbfadbd1e21a710f4de0a3bd05ef0
Jun 21 07:51:33 dist-test-slave-lzx3 krb5kdc[23388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.253@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.192.252@KRBTEST.COM; defaulting to no policy
I20260621 07:51:33.748235 23472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:33.738111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35996 (local address 127.0.192.254:34805)
0621 07:51:33.738230 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:33.738234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:33.738893 (+ 659us) server_negotiation.cc:407] Connection header received
0621 07:51:33.739692 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:33.739695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:33.739750 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:33.739848 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:33.740939 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:33.741524 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:33.742501 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:33.742738 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:33.742839 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:33.745010 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:33.745013 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:33.745016 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:33.745043 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:33.746269 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:33.746760 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:33.746765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:33.746766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:33.746844 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:33.747173 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:33.747178 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:33.747180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:33.747344 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:33.747450 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:33.747913 (+ 463us) server_negotiation.cc:299] Negotiation successful
0621 07:51:33.748057 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260621 07:51:33.748909 23546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:33.738398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:34805 (local address 127.0.0.1:35996)
0621 07:51:33.738757 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:33.738792 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:33.739485 (+ 693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:33.739984 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:33.739990 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:33.740257 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:33.740805 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:33.740821 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:33.741640 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:33.741643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:33.742381 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:33.742389 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:33.742868 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:33.742871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:33.743028 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:33.743480 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:33.743493 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:33.744868 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:33.746375 (+ 1507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:33.746380 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:33.746391 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:33.746640 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:33.746942 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:33.746945 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:33.746947 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:33.747078 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:33.747446 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:33.747452 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:33.747675 (+ 223us) client_negotiation.cc:769] Sending connection context
0621 07:51:33.747842 (+ 167us) client_negotiation.cc:240] Negotiation successful
0621 07:51:33.748152 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":96,"threads_started":1}
I20260621 07:51:33.750087 23544 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.192.253" port: 32909 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.192.252@KRBTEST.COM" created.
Jun 21 07:51:33 dist-test-slave-lzx3 krb5kdc[23388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.253@KRBTEST.COM for kudu/127.0.192.253@KRBTEST.COM
I20260621 07:51:33.758514 23546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:33.750536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.253:32909 (local address 127.0.0.1:46316)
0621 07:51:33.750644 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:33.750656 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:33.750727 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:33.751118 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:33.751120 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:33.751265 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:33.751342 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:33.751346 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:33.752223 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:33.752226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:33.752806 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:33.752814 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:33.753375 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:33.753379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:33.753531 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:33.753948 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:33.753961 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:33.755247 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:33.757092 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:33.757097 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:33.757099 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:33.757282 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:33.757571 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:33.757573 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:33.757575 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:33.757653 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:33.758072 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:33.758074 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:33.758144 (+ 70us) client_negotiation.cc:769] Sending connection context
0621 07:51:33.758250 (+ 106us) client_negotiation.cc:240] Negotiation successful
0621 07:51:33.758357 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21,"mutex_wait_us":37}
I20260621 07:51:33.758610 23548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:33.750587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46316 (local address 127.0.192.253:32909)
0621 07:51:33.750834 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:33.750846 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:33.750868 (+ 22us) server_negotiation.cc:407] Connection header received
0621 07:51:33.750894 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:33.750897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:33.751015 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:33.751137 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:33.751441 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:33.752111 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:33.752906 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:33.753237 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:33.753379 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:33.755369 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:33.755379 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:33.755393 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:33.755410 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:33.756966 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:33.757389 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:33.757395 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:33.757396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:33.757472 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:33.757748 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:33.757751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:33.757753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:33.757972 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:33.758101 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:33.758328 (+ 227us) server_negotiation.cc:299] Negotiation successful
0621 07:51:33.758431 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":169,"thread_start_us":72,"threads_started":1}
I20260621 07:51:33.759943 23544 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.192.252" port: 44291 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:51:33.760462 23546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:33.760279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:44291 (local address 127.0.0.1:58608)
0621 07:51:33.760362 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:33.760416 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.252:44291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260621 07:51:33.760685 23492 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.192.252:44291: connect: Connection refused (error 111)
I20260621 07:51:33.761304 23546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:33.761145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:44291 (local address 127.0.0.1:58616)
0621 07:51:33.761225 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:33.761262 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.252:44291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260621 07:51:33.761439 23544 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.192.252:44291: Network error: Client connection negotiation failed: client connection to 127.0.192.252:44291: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.192.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/krb5kdc/kudu_127.0.192.252.keytab.
Entry for principal kudu/127.0.192.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/krb5kdc/kudu_127.0.192.252.keytab.
I20260621 07:51:33.762897 771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
/tmp/dist-test-taskeKA8lE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.252:44291
--webserver_interface=127.0.192.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/krb5kdc/kudu_127.0.192.252.keytab
--principal=kudu/127.0.192.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.192.212:41147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.192.254:34805,127.0.192.253:32909,127.0.192.252:44291
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:51:33.765599 23469 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.192.253" port: 32909 } attempt: 3
Jun 21 07:51:33 dist-test-slave-lzx3 krb5kdc[23388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for kudu/127.0.192.253@KRBTEST.COM
I20260621 07:51:33.772675 23471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:33.765962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.253:32909 (local address 127.0.0.1:46320)
0621 07:51:33.766052 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:33.766083 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:33.766165 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:33.766346 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:33.766349 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:33.766489 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:33.766565 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:33.766570 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:33.767270 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:33.767274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:33.767808 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:33.767816 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:33.768282 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:33.768285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:33.768380 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:33.768762 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:33.768773 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:33.769976 (+ 1203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:33.771420 (+ 1444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:33.771423 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:33.771425 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:33.771594 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:33.771837 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:33.771840 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:33.771842 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:33.771915 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:33.772228 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:33.772230 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:33.772309 (+ 79us) client_negotiation.cc:769] Sending connection context
0621 07:51:33.772404 (+ 95us) client_negotiation.cc:240] Negotiation successful
0621 07:51:33.772514 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260621 07:51:33.772673 23548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:33.766020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46320 (local address 127.0.192.253:32909)
0621 07:51:33.766148 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:33.766151 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:33.766174 (+ 23us) server_negotiation.cc:407] Connection header received
0621 07:51:33.766231 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:33.766234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:33.766265 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:33.766325 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:33.766665 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:33.767161 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:33.767963 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:33.768175 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:33.768278 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:33.770102 (+ 1824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:33.770105 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:33.770107 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:33.770127 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:33.771308 (+ 1181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:33.771683 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:33.771686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:33.771688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:33.771756 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:33.772000 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:33.772003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:33.772004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:33.772142 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:33.772232 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:33.772457 (+ 225us) server_negotiation.cc:299] Negotiation successful
0621 07:51:33.772572 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 07:51:33.773854 23469 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.192.252" port: 44291 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:51:33.774870 23471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:33.774690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:44291 (local address 127.0.0.1:58620)
0621 07:51:33.774783 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:33.774821 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.252:44291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260621 07:51:33.775430 23471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:33.775240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:44291 (local address 127.0.0.1:58624)
0621 07:51:33.775354 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:33.775389 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.252:44291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260621 07:51:33.775573 23469 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.192.252:44291: Network error: Client connection negotiation failed: client connection to 127.0.192.252:44291: connect: Connection refused (error 111)
I20260621 07:51:33.806736 23469 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.192.252" port: 44291 } attempt: 1
I20260621 07:51:33.807318 23471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:33.807106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:44291 (local address 127.0.0.1:58632)
0621 07:51:33.807217 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:33.807264 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.252:44291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260621 07:51:33.807994 23471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:33.807776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:44291 (local address 127.0.0.1:58634)
0621 07:51:33.807876 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:33.807943 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.252:44291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260621 07:51:33.808105 23469 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.192.252:44291: Network error: Client connection negotiation failed: client connection to 127.0.192.252:44291: connect: Connection refused (error 111)
I20260621 07:51:33.810622 23544 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.192.252" port: 44291 } attempt: 1
I20260621 07:51:33.811159 23546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:33.810956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:44291 (local address 127.0.0.1:58650)
0621 07:51:33.811060 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:33.811110 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.252:44291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260621 07:51:33.811772 23546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:33.811552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:44291 (local address 127.0.0.1:58660)
0621 07:51:33.811671 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:33.811729 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.252:44291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260621 07:51:33.811885 23544 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.192.252:44291: Network error: Client connection negotiation failed: client connection to 127.0.192.252:44291: connect: Connection refused (error 111)
W20260621 07:51:33.861158 23551 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 07:51:33.861413 23551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:51:33.861480 23551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:51:33.861555 23551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:51:33.865180 23551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:51:33.865244 23551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:51:33.865295 23551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:51:33.865355 23551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:51:33.865402 23551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:51:33.869946 23551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.192.212:41147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.192.254:34805,127.0.192.253:32909,127.0.192.252:44291
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/krb5kdc/kudu_127.0.192.252.keytab
--principal=kudu/127.0.192.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.192.252:44291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.192.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Jun 2026 07:00:36 UTC on ccycloud.quasar-apkvod.root.comops.site
build id 4457933
I20260621 07:51:33.870777 23551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:51:33.871610 23551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:51:33.879137 23556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:51:33.879130 23559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:33.879266 23551 server_base.cc:1034] running on GCE node
W20260621 07:51:33.879397 23557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:51:33.879855 23551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:51:33.880029 23544 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.192.252" port: 44291 } attempt: 2
I20260621 07:51:33.880728 23546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:33.880453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:44291 (local address 127.0.0.1:58664)
0621 07:51:33.880617 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:33.880663 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.252:44291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260621 07:51:33.880998 23551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:51:33.881409 23546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:33.881188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:44291 (local address 127.0.0.1:58674)
0621 07:51:33.881308 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:33.881353 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.252:44291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260621 07:51:33.881590 23544 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.192.252:44291: Network error: Client connection negotiation failed: client connection to 127.0.192.252:44291: connect: Connection refused (error 111)
I20260621 07:51:33.882164 23551 hybrid_clock.cc:648] HybridClock initialized: now 1782028293882143 us; error 31 us; skew 500 ppm
I20260621 07:51:33.883244 23469 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.192.252" port: 44291 } attempt: 2
I20260621 07:51:33.883795 23471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:33.883582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:44291 (local address 127.0.0.1:58688)
0621 07:51:33.883689 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:33.883739 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.252:44291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
Jun 21 07:51:33 dist-test-slave-lzx3 krb5kdc[23388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:51:33.884371 23471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:33.884188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:44291 (local address 127.0.0.1:58702)
0621 07:51:33.884282 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:33.884326 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.192.252:44291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260621 07:51:33.884473 23469 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.192.252:44291: Network error: Client connection negotiation failed: client connection to 127.0.192.252:44291: connect: Connection refused (error 111)
I20260621 07:51:33.884613 23551 init.cc:377] Logged in from keytab as kudu/127.0.192.252@KRBTEST.COM (short username kudu)
I20260621 07:51:33.885372 23551 webserver.cc:460] Webserver started at http://127.0.192.252:38613/ using document root <none> and password file <none>
I20260621 07:51:33.885839 23551 fs_manager.cc:359] Metadata directory not provided
I20260621 07:51:33.885918 23551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:51:33.886094 23551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:51:33.888003 23551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-2/data/instance:
uuid: "7c3bd96434ac476c875a16578556b4f2"
format_stamp: "Formatted at 2026-06-21 07:51:33 on dist-test-slave-lzx3"
server_key: "d30acda979b2175421ffac5d0bfad17a"
server_key_iv: "d292b5a3d3de6058d26cb22f9e0ead80"
server_key_version: "encryptionkey@0"
I20260621 07:51:33.888510 23551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-2/wal/instance:
uuid: "7c3bd96434ac476c875a16578556b4f2"
format_stamp: "Formatted at 2026-06-21 07:51:33 on dist-test-slave-lzx3"
server_key: "d30acda979b2175421ffac5d0bfad17a"
server_key_iv: "d292b5a3d3de6058d26cb22f9e0ead80"
server_key_version: "encryptionkey@0"
I20260621 07:51:33.891489 23551 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260621 07:51:33.893638 23566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:51:33.894757 23551 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 07:51:33.894886 23551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-2/data,/tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-2/wal
uuid: "7c3bd96434ac476c875a16578556b4f2"
format_stamp: "Formatted at 2026-06-21 07:51:33 on dist-test-slave-lzx3"
server_key: "d30acda979b2175421ffac5d0bfad17a"
server_key_iv: "d292b5a3d3de6058d26cb22f9e0ead80"
server_key_version: "encryptionkey@0"
I20260621 07:51:33.894979 23551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:51:33.913322 23551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:51:33.915697 23551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:51:33.915867 23551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:51:33.922355 23551 rpc_server.cc:307] RPC server started. Bound to: 127.0.192.252:44291
I20260621 07:51:33.922560 23618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.192.252:44291 every 8 connection(s)
I20260621 07:51:33.923219 23551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-2/data/info.pb
I20260621 07:51:33.925626 23619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:51:33.927388 771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu as pid 23551
I20260621 07:51:33.927529 771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeKA8lE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782028028522580-771-0/minicluster-data/master-2/wal/instance
I20260621 07:51:33.927826 771 external_mini_cluster.cc:1428] Setting key f920e78353983d7e0bd5867721d0fb50
I20260621 07:51:33.929421 23619 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34805 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 07:51:33 dist-test-slave-lzx3 krb5kdc[23388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.252@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
Jun 21 07:51:33 dist-test-slave-lzx3 krb5kdc[23388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.192.252@KRBTEST.COM
I20260621 07:51:33.939934 23472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:33.931445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36000 (local address 127.0.192.254:34805)
0621 07:51:33.931567 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:33.931571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:33.931781 (+ 210us) server_negotiation.cc:407] Connection header received
0621 07:51:33.931903 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:33.931908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:33.931957 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:33.932021 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:33.932782 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:33.933243 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:33.934127 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:33.934341 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:33.934468 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:33.936534 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:33.936537 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:33.936539 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:33.936597 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:33.938124 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:33.938599 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:33.938602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:33.938604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:33.938678 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:33.938995 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:33.938998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:33.938999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:33.939164 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:33.939263 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:33.939674 (+ 411us) server_negotiation.cc:299] Negotiation successful
0621 07:51:33.939778 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 07:51:33.940716 23623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:33.931316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.254:34805 (local address 127.0.0.1:36000)
0621 07:51:33.931664 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:33.931705 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:33.931812 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:33.932091 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:33.932096 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:33.932474 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:33.932659 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:33.932670 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:33.933364 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:33.933367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:33.934004 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:33.934013 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:33.934500 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:33.934504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:33.934606 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:33.934965 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:33.934978 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:33.936351 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:33.938226 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:33.938236 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:33.938243 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:33.938502 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:33.938779 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:33.938782 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:33.938784 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:33.938899 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:33.939261 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:33.939267 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:33.939468 (+ 201us) client_negotiation.cc:769] Sending connection context
0621 07:51:33.939635 (+ 167us) client_negotiation.cc:240] Negotiation successful
0621 07:51:33.939878 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":111,"threads_started":1}
I20260621 07:51:33.940718 23622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:33.929032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58718 (local address 127.0.192.252:44291)
0621 07:51:33.929364 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:33.929394 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:33.929424 (+ 30us) server_negotiation.cc:407] Connection header received
0621 07:51:33.930046 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:33.930061 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:33.930385 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:33.930629 (+ 244us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:33.931438 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:33.932763 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:33.933611 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:33.934051 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:33.934213 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:33.937005 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:33.937013 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:33.937025 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:33.937054 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:33.938636 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:33.939036 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:33.939041 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:33.939042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:33.939119 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:33.939380 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:33.939384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:33.939386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:33.939589 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:33.939777 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:33.939943 (+ 166us) server_negotiation.cc:299] Negotiation successful
0621 07:51:33.940085 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":113,"threads_started":1}
I20260621 07:51:33.941993 23619 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.192.253" port: 32909 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 07:51:33 dist-test-slave-lzx3 krb5kdc[23388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.252@KRBTEST.COM for kudu/127.0.192.253@KRBTEST.COM
Jun 21 07:51:33 dist-test-slave-lzx3 krb5kdc[23388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.192.254@KRBTEST.COM
I20260621 07:51:33.949957 23623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:33.942377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.253:32909 (local address 127.0.0.1:46334)
0621 07:51:33.942462 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:33.942474 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:33.942567 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:33.942990 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:33.942992 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:33.943136 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:33.943236 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:33.943243 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:33.944004 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:33.944007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:33.944627 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:33.944636 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:33.945065 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:33.945068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:33.945156 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:33.945534 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:33.945546 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:33.946556 (+ 1010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:33.948522 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:33.948526 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:33.948529 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:33.948767 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:33.949064 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:33.949066 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:33.949067 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:33.949131 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:33.949524 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:33.949527 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:33.949611 (+ 84us) client_negotiation.cc:769] Sending connection context
0621 07:51:33.949718 (+ 107us) client_negotiation.cc:240] Negotiation successful
0621 07:51:33.949838 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260621 07:51:33.950065 23548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:33.942496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46334 (local address 127.0.192.253:32909)
0621 07:51:33.942673 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:33.942676 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:33.942688 (+ 12us) server_negotiation.cc:407] Connection header received
0621 07:51:33.942744 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:33.942746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:33.942804 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:33.942875 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:33.943351 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:33.943905 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:33.944744 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:33.944959 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:33.945033 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:33.946713 (+ 1680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:33.946720 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:33.946722 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:33.946758 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:33.948386 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:33.948883 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:33.948889 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:33.948892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:33.948973 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:33.949213 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:33.949216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:33.949218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:33.949416 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:33.949515 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:33.949787 (+ 272us) server_negotiation.cc:299] Negotiation successful
0621 07:51:33.949904 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260621 07:51:33.950678 23619 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.192.252" port: 44291 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:51:33.951139 23472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:33.942851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36006 (local address 127.0.192.254:34805)
0621 07:51:33.942935 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:33.942938 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:33.942948 (+ 10us) server_negotiation.cc:407] Connection header received
0621 07:51:33.943056 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:33.943058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:33.943084 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:33.943135 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:33.943680 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:33.944212 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:33.945107 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:33.945381 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:33.945513 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:33.947858 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:33.947865 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:33.947870 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:33.947906 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:33.949367 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:33.949783 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:33.949786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:33.949788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:33.949864 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:33.950289 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:33.950292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:33.950294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:33.950508 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:33.950668 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:33.950880 (+ 212us) server_negotiation.cc:299] Negotiation successful
0621 07:51:33.951009 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
Jun 21 07:51:33 dist-test-slave-lzx3 krb5kdc[23388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.252@KRBTEST.COM for kudu/127.0.192.252@KRBTEST.COM
Jun 21 07:51:33 dist-test-slave-lzx3 krb5kdc[23388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.192.253@KRBTEST.COM
I20260621 07:51:33.958528 23623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:33.951065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:44291 (local address 127.0.0.1:58730)
0621 07:51:33.951162 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:33.951172 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:33.951263 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:33.951556 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:33.951561 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:33.951713 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:33.951771 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:33.951776 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:33.952643 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:33.952646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:33.953263 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:33.953272 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:33.953691 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:33.953697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:33.953795 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:33.954182 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:33.954192 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:33.955469 (+ 1277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:33.957181 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:33.957183 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:33.957185 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:33.957358 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:33.957638 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:33.957640 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:33.957642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:33.957716 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:33.958044 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:33.958047 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:33.958118 (+ 71us) client_negotiation.cc:769] Sending connection context
0621 07:51:33.958237 (+ 119us) client_negotiation.cc:240] Negotiation successful
0621 07:51:33.958361 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":19}
I20260621 07:51:33.958545 23622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:33.951319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58730 (local address 127.0.192.252:44291)
0621 07:51:33.951391 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:33.951394 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:33.951403 (+ 9us) server_negotiation.cc:407] Connection header received
0621 07:51:33.951432 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:33.951433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:33.951460 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:33.951509 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:33.951877 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:33.952531 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:33.953373 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:33.953585 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:33.953685 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:33.955597 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:33.955601 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:33.955606 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:33.955632 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:33.957078 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:33.957468 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:33.957473 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:33.957479 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:33.957542 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:33.957801 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:33.957806 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:33.957808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:33.957958 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:33.958052 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:33.958292 (+ 240us) server_negotiation.cc:299] Negotiation successful
0621 07:51:33.958409 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":24}
I20260621 07:51:33.960170 23548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:33.952043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46350 (local address 127.0.192.253:32909)
0621 07:51:33.952134 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:33.952137 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:33.952155 (+ 18us) server_negotiation.cc:407] Connection header received
0621 07:51:33.952228 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:33.952230 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:33.952284 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:33.952353 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:33.952957 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:33.953591 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:33.954503 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:33.954754 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:33.954827 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:33.957084 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:33.957088 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:33.957090 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:33.957115 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:33.958513 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:33.959159 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:33.959161 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:33.959163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:33.959235 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:33.959475 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:33.959478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:33.959479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:33.959670 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:33.959862 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:33.959926 (+ 64us) server_negotiation.cc:299] Negotiation successful
0621 07:51:33.960027 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260621 07:51:33.961827 23619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c3bd96434ac476c875a16578556b4f2: Bootstrap starting.
I20260621 07:51:33.963224 771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:51:33.964010 23619 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c3bd96434ac476c875a16578556b4f2: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:33.964767 23619 log.cc:824] T 00000000000000000000000000000000 P 7c3bd96434ac476c875a16578556b4f2: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:33.964309 23443 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "371e06a939234fd59161dfbd663a8351"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36006
I20260621 07:51:33.966374 23619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c3bd96434ac476c875a16578556b4f2: No bootstrap required, opened a new log
I20260621 07:51:33.969120 23619 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c3bd96434ac476c875a16578556b4f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371e06a939234fd59161dfbd663a8351" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34805 } } peers { permanent_uuid: "942def5184454e568c77a7b32b42cffc" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 32909 } } peers { permanent_uuid: "7c3bd96434ac476c875a16578556b4f2" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 44291 } }
I20260621 07:51:33.969276 23619 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c3bd96434ac476c875a16578556b4f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c3bd96434ac476c875a16578556b4f2, State: Initialized, Role: FOLLOWER
I20260621 07:51:33.969571 23619 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c3bd96434ac476c875a16578556b4f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371e06a939234fd59161dfbd663a8351" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34805 } } peers { permanent_uuid: "942def5184454e568c77a7b32b42cffc" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 32909 } } peers { permanent_uuid: "7c3bd96434ac476c875a16578556b4f2" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 44291 } }
I20260621 07:51:33.969976 23625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c3bd96434ac476c875a16578556b4f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371e06a939234fd59161dfbd663a8351" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34805 } } peers { permanent_uuid: "942def5184454e568c77a7b32b42cffc" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 32909 } } peers { permanent_uuid: "7c3bd96434ac476c875a16578556b4f2" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 44291 } } }
I20260621 07:51:33.970103 23625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c3bd96434ac476c875a16578556b4f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:51:33.970824 23619 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c3bd96434ac476c875a16578556b4f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:51:33.972740 23544 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.192.252" port: 44291 } attempt: 3
W20260621 07:51:33.975656 23636 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7c3bd96434ac476c875a16578556b4f2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:51:33.975723 23636 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:51:33.974278 23551 master_runner.cc:186] Error getting master registration for 127.0.192.254:34805: OK, instance_id { permanent_uuid: "371e06a939234fd59161dfbd663a8351" instance_seqno: 1782028293527495 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 07:51:33.977582 23551 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Jun 21 07:51:33 dist-test-slave-lzx3 krb5kdc[23388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.253@KRBTEST.COM for kudu/127.0.192.252@KRBTEST.COM
I20260621 07:51:33.982008 23546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:33.973075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:44291 (local address 127.0.0.1:58740)
0621 07:51:33.973397 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:33.973412 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:33.973513 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:33.973747 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:33.973750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:33.973923 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:33.974028 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:33.974035 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:33.974959 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:33.974962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:33.975776 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:33.975789 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:33.976535 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:33.976538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:33.976658 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:33.977136 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:33.977150 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:33.978626 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:33.980489 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:33.980492 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:33.980494 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:33.980760 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:33.981027 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:33.981030 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:33.981032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:33.981121 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:33.981556 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:33.981559 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:33.981646 (+ 87us) client_negotiation.cc:769] Sending connection context
0621 07:51:33.981749 (+ 103us) client_negotiation.cc:240] Negotiation successful
0621 07:51:33.981864 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250}
I20260621 07:51:33.982033 23622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:33.973299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58740 (local address 127.0.192.252:44291)
0621 07:51:33.973383 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:33.973387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:33.973468 (+ 81us) server_negotiation.cc:407] Connection header received
0621 07:51:33.973603 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:33.973606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:33.973646 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:33.973728 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:33.974124 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:33.974831 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:33.975900 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:33.976309 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:33.976418 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:33.978749 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:33.978753 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:33.978755 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:33.978776 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:33.980369 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:33.980848 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:33.980851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:33.980853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:33.980928 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:33.981225 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:33.981228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:33.981230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:33.981446 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:33.981546 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:33.981801 (+ 255us) server_negotiation.cc:299] Negotiation successful
0621 07:51:33.981898 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260621 07:51:33.983611 23469 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.192.252" port: 44291 } attempt: 3
I20260621 07:51:33.984934 23544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 942def5184454e568c77a7b32b42cffc: Bootstrap starting.
I20260621 07:51:33.987571 23544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 942def5184454e568c77a7b32b42cffc: Neither blocks nor log segments found. Creating new log.
Jun 21 07:51:33 dist-test-slave-lzx3 krb5kdc[23388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782028293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.192.254@KRBTEST.COM for kudu/127.0.192.252@KRBTEST.COM
I20260621 07:51:33.988373 23544 log.cc:824] T 00000000000000000000000000000000 P 942def5184454e568c77a7b32b42cffc: Log is configured to *not* fsync() on all Append() calls
I20260621 07:51:33.990244 23544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 942def5184454e568c77a7b32b42cffc: No bootstrap required, opened a new log
I20260621 07:51:33.991976 23471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:33.983957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.192.252:44291 (local address 127.0.0.1:58750)
0621 07:51:33.984066 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0621 07:51:33.984082 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:51:33.984151 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:51:33.984625 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:51:33.984630 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:51:33.984799 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:51:33.984872 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:33.984877 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:33.985882 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:33.985885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:33.986377 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:51:33.986384 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:51:33.986930 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:51:33.986934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:51:33.987026 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:33.987456 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:51:33.987489 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:51:33.988779 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:51:33.990614 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:33.990617 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:33.990619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:33.990804 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:33.991054 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:51:33.991056 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:51:33.991058 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:51:33.991133 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:51:33.991517 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:51:33.991520 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:51:33.991605 (+ 85us) client_negotiation.cc:769] Sending connection context
0621 07:51:33.991698 (+ 93us) client_negotiation.cc:240] Negotiation successful
0621 07:51:33.991820 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260621 07:51:33.992542 23622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:33.984277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58750 (local address 127.0.192.252:44291)
0621 07:51:33.984383 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:33.984387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:33.984398 (+ 11us) server_negotiation.cc:407] Connection header received
0621 07:51:33.984431 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:33.984433 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:33.984476 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:33.984550 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:33.984992 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:33.985769 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:33.986499 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:33.986836 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:33.986926 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:33.988919 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:33.988922 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:33.988925 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:33.988946 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:33.990514 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:33.990899 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:33.990904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:33.990905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:33.990980 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:33.991217 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:33.991222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:33.991223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:33.991398 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:33.992215 (+ 817us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:33.992287 (+ 72us) server_negotiation.cc:299] Negotiation successful
0621 07:51:33.992377 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260621 07:51:33.993906 23544 raft_consensus.cc:348] T 00000000000000000000000000000000 P 942def5184454e568c77a7b32b42cffc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371e06a939234fd59161dfbd663a8351" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34805 } } peers { permanent_uuid: "942def5184454e568c77a7b32b42cffc" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 32909 } } peers { permanent_uuid: "7c3bd96434ac476c875a16578556b4f2" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 44291 } }
I20260621 07:51:33.994110 23544 raft_consensus.cc:729] T 00000000000000000000000000000000 P 942def5184454e568c77a7b32b42cffc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 942def5184454e568c77a7b32b42cffc, State: Initialized, Role: FOLLOWER
I20260621 07:51:33.994452 23544 consensus_queue.cc:260] T 00000000000000000000000000000000 P 942def5184454e568c77a7b32b42cffc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371e06a939234fd59161dfbd663a8351" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34805 } } peers { permanent_uuid: "942def5184454e568c77a7b32b42cffc" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 32909 } } peers { permanent_uuid: "7c3bd96434ac476c875a16578556b4f2" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 44291 } }
I20260621 07:51:33.995033 23639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 942def5184454e568c77a7b32b42cffc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371e06a939234fd59161dfbd663a8351" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34805 } } peers { permanent_uuid: "942def5184454e568c77a7b32b42cffc" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 32909 } } peers { permanent_uuid: "7c3bd96434ac476c875a16578556b4f2" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 44291 } } }
I20260621 07:51:33.995247 23639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 942def5184454e568c77a7b32b42cffc [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:51:33.995621 23469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 371e06a939234fd59161dfbd663a8351: Bootstrap starting.
I20260621 07:51:33.995674 23544 sys_catalog.cc:564] T 00000000000000000000000000000000 P 942def5184454e568c77a7b32b42cffc [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:51:33.998625 23469 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 371e06a939234fd59161dfbd663a8351: Neither blocks nor log segments found. Creating new log.
I20260621 07:51:33.999589 23469 log.cc:824] T 00000000000000000000000000000000 P 371e06a939234fd59161dfbd663a8351: Log is configured to *not* fsync() on all Append() calls
W20260621 07:51:34.000692 23650 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 942def5184454e568c77a7b32b42cffc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:51:34.000810 23650 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:51:34.001420 23469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 371e06a939234fd59161dfbd663a8351: No bootstrap required, opened a new log
I20260621 07:51:33.998890 23476 master_runner.cc:186] Error getting master registration for 127.0.192.254:34805: OK, instance_id { permanent_uuid: "371e06a939234fd59161dfbd663a8351" instance_seqno: 1782028293527495 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 07:51:34.002528 23476 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 07:51:34.003727 23469 raft_consensus.cc:348] T 00000000000000000000000000000000 P 371e06a939234fd59161dfbd663a8351 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371e06a939234fd59161dfbd663a8351" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34805 } } peers { permanent_uuid: "942def5184454e568c77a7b32b42cffc" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 32909 } } peers { permanent_uuid: "7c3bd96434ac476c875a16578556b4f2" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 44291 } }
I20260621 07:51:34.003922 23469 raft_consensus.cc:729] T 00000000000000000000000000000000 P 371e06a939234fd59161dfbd663a8351 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371e06a939234fd59161dfbd663a8351, State: Initialized, Role: FOLLOWER
I20260621 07:51:34.004241 23469 consensus_queue.cc:260] T 00000000000000000000000000000000 P 371e06a939234fd59161dfbd663a8351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371e06a939234fd59161dfbd663a8351" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34805 } } peers { permanent_uuid: "942def5184454e568c77a7b32b42cffc" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 32909 } } peers { permanent_uuid: "7c3bd96434ac476c875a16578556b4f2" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 44291 } }
I20260621 07:51:34.004683 23653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 371e06a939234fd59161dfbd663a8351 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371e06a939234fd59161dfbd663a8351" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34805 } } peers { permanent_uuid: "942def5184454e568c77a7b32b42cffc" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 32909 } } peers { permanent_uuid: "7c3bd96434ac476c875a16578556b4f2" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 44291 } } }
I20260621 07:51:34.004825 23653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 371e06a939234fd59161dfbd663a8351 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:51:34.005373 23469 sys_catalog.cc:564] T 00000000000000000000000000000000 P 371e06a939234fd59161dfbd663a8351 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:51:34.005426 23443 raft_consensus.cc:482] T 00000000000000000000000000000000 P 371e06a939234fd59161dfbd663a8351 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260621 07:51:34.005551 23443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 371e06a939234fd59161dfbd663a8351 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:34.006630 23443 raft_consensus.cc:504] T 00000000000000000000000000000000 P 371e06a939234fd59161dfbd663a8351 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371e06a939234fd59161dfbd663a8351" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34805 } } peers { permanent_uuid: "942def5184454e568c77a7b32b42cffc" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 32909 } } peers { permanent_uuid: "7c3bd96434ac476c875a16578556b4f2" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 44291 } }
I20260621 07:51:34.008025 23593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "371e06a939234fd59161dfbd663a8351" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7c3bd96434ac476c875a16578556b4f2"
I20260621 07:51:34.008224 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "371e06a939234fd59161dfbd663a8351" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "942def5184454e568c77a7b32b42cffc"
I20260621 07:51:34.008280 23593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c3bd96434ac476c875a16578556b4f2 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:34.008402 23518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 942def5184454e568c77a7b32b42cffc [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:51:34.007460 23443 leader_election.cc:288] T 00000000000000000000000000000000 P 371e06a939234fd59161dfbd663a8351 [CANDIDATE]: Term 1 election: Requested vote from peers 942def5184454e568c77a7b32b42cffc (127.0.192.253:32909), 7c3bd96434ac476c875a16578556b4f2 (127.0.192.252:44291)
I20260621 07:51:34.009603 23593 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7c3bd96434ac476c875a16578556b4f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371e06a939234fd59161dfbd663a8351 in term 1.
I20260621 07:51:34.009610 23518 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 942def5184454e568c77a7b32b42cffc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371e06a939234fd59161dfbd663a8351 in term 1.
I20260621 07:51:34.010103 23417 leader_election.cc:302] T 00000000000000000000000000000000 P 371e06a939234fd59161dfbd663a8351 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371e06a939234fd59161dfbd663a8351, 7c3bd96434ac476c875a16578556b4f2; no voters:
I20260621 07:51:34.010349 23653 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 371e06a939234fd59161dfbd663a8351 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:51:34.010524 23653 raft_consensus.cc:686] T 00000000000000000000000000000000 P 371e06a939234fd59161dfbd663a8351 [term 1 LEADER]: Becoming Leader. State: Replica: 371e06a939234fd59161dfbd663a8351, State: Running, Role: LEADER
I20260621 07:51:34.010812 23653 consensus_queue.cc:237] T 00000000000000000000000000000000 P 371e06a939234fd59161dfbd663a8351 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371e06a939234fd59161dfbd663a8351" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34805 } } peers { permanent_uuid: "942def5184454e568c77a7b32b42cffc" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 32909 } } peers { permanent_uuid: "7c3bd96434ac476c875a16578556b4f2" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 44291 } }
I20260621 07:51:34.012427 23653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 371e06a939234fd59161dfbd663a8351 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 371e06a939234fd59161dfbd663a8351. Latest consensus state: current_term: 1 leader_uuid: "371e06a939234fd59161dfbd663a8351" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371e06a939234fd59161dfbd663a8351" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34805 } } peers { permanent_uuid: "942def5184454e568c77a7b32b42cffc" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 32909 } } peers { permanent_uuid: "7c3bd96434ac476c875a16578556b4f2" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 44291 } } }
I20260621 07:51:34.012534 23653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 371e06a939234fd59161dfbd663a8351 [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:34.015516 23401 master_runner.cc:263] No leader master found from master 371e06a939234fd59161dfbd663a8351
I20260621 07:51:34.015578 23401 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260621 07:51:34.015861 23664 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 371e06a939234fd59161dfbd663a8351: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:51:34.015974 23664 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:51:34.016088 23671 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:51:34.017007 23671 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:51:34.023553 23593 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7c3bd96434ac476c875a16578556b4f2 [term 1 FOLLOWER]: Refusing update from remote peer 371e06a939234fd59161dfbd663a8351: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:51:34.024224 23518 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 942def5184454e568c77a7b32b42cffc [term 1 FOLLOWER]: Refusing update from remote peer 371e06a939234fd59161dfbd663a8351: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260621 07:51:34.024427 23422 tablet.cc:2369] T 00000000000000000000000000000000 P 371e06a939234fd59161dfbd663a8351: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:51:34.024943 23653 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 371e06a939234fd59161dfbd663a8351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "942def5184454e568c77a7b32b42cffc" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:51:34.025278 23653 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 371e06a939234fd59161dfbd663a8351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c3bd96434ac476c875a16578556b4f2" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 44291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:51:34.028995 23639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 942def5184454e568c77a7b32b42cffc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 371e06a939234fd59161dfbd663a8351. Latest consensus state: current_term: 1 leader_uuid: "371e06a939234fd59161dfbd663a8351" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371e06a939234fd59161dfbd663a8351" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34805 } } peers { permanent_uuid: "942def5184454e568c77a7b32b42cffc" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 32909 } } peers { permanent_uuid: "7c3bd96434ac476c875a16578556b4f2" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 44291 } } }
I20260621 07:51:34.029102 23639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 942def5184454e568c77a7b32b42cffc [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:51:34.031599 23625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c3bd96434ac476c875a16578556b4f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 371e06a939234fd59161dfbd663a8351. Latest consensus state: current_term: 1 leader_uuid: "371e06a939234fd59161dfbd663a8351" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371e06a939234fd59161dfbd663a8351" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34805 } } peers { permanent_uuid: "942def5184454e568c77a7b32b42cffc" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 32909 } } peers { permanent_uuid: "7c3bd96434ac476c875a16578556b4f2" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 44291 } } }
I20260621 07:51:34.031708 23625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c3bd96434ac476c875a16578556b4f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:51:34.035650 23653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 371e06a939234fd59161dfbd663a8351 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "371e06a939234fd59161dfbd663a8351" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371e06a939234fd59161dfbd663a8351" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34805 } } peers { permanent_uuid: "942def5184454e568c77a7b32b42cffc" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 32909 } } peers { permanent_uuid: "7c3bd96434ac476c875a16578556b4f2" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 44291 } } }
I20260621 07:51:34.035771 23653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 371e06a939234fd59161dfbd663a8351 [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:34.036850 23639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 942def5184454e568c77a7b32b42cffc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "371e06a939234fd59161dfbd663a8351" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371e06a939234fd59161dfbd663a8351" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34805 } } peers { permanent_uuid: "942def5184454e568c77a7b32b42cffc" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 32909 } } peers { permanent_uuid: "7c3bd96434ac476c875a16578556b4f2" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 44291 } } }
I20260621 07:51:34.036955 23639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 942def5184454e568c77a7b32b42cffc [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:51:34.037223 23676 mvcc.cc:205] Tried to move back new op lower bound from 7299187892312821760 to 7299187892271783936. Current Snapshot: MvccSnapshot[applied={T|T < 7299187892312821760}]
I20260621 07:51:34.037379 23677 mvcc.cc:205] Tried to move back new op lower bound from 7299187892312821760 to 7299187892271783936. Current Snapshot: MvccSnapshot[applied={T|T < 7299187892312821760}]
I20260621 07:51:34.043076 23472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:34.016527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36008 (local address 127.0.192.254:34805)
0621 07:51:34.016629 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:34.016633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:34.031555 (+ 14922us) server_negotiation.cc:407] Connection header received
0621 07:51:34.031596 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:34.031599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:34.031641 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:34.031711 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:34.032981 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:34.033710 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:34.034740 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:34.035032 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:34.035109 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:34.037540 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:34.037544 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:34.037547 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:34.037571 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:34.039456 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:34.041340 (+ 1884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:34.041343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:34.041345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:34.041414 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:34.042055 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:34.042058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:34.042060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:34.042242 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:34.042333 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:34.042803 (+ 470us) server_negotiation.cc:299] Negotiation successful
0621 07:51:34.042912 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260621 07:51:34.044462 23548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:34.015947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46356 (local address 127.0.192.253:32909)
0621 07:51:34.016055 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:34.016060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:34.027983 (+ 11923us) server_negotiation.cc:407] Connection header received
0621 07:51:34.028169 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:34.028172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:34.028216 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:34.028262 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:34.029258 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:34.029927 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:34.031089 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:34.031394 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:34.031465 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:34.037953 (+ 6488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:34.037958 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:34.037970 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:34.037993 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:34.040202 (+ 2209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:34.040972 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:34.040976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:34.040977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:34.041049 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:34.041681 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:34.041685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:34.041686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:34.041870 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:34.041983 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:34.044222 (+ 2239us) server_negotiation.cc:299] Negotiation successful
0621 07:51:34.044326 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260621 07:51:34.046875 23671 catalog_manager.cc:1347] Generated new cluster ID: 6d784cd81df042acbed7af9c419c1e60
I20260621 07:51:34.046931 23671 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:51:34.051918 23622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:34.017092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58758 (local address 127.0.192.252:44291)
0621 07:51:34.017198 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:34.017203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:34.037955 (+ 20752us) server_negotiation.cc:407] Connection header received
0621 07:51:34.038075 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:34.038082 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:34.038132 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:34.038205 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:34.044461 (+ 6256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:34.045188 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:34.046444 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:34.046789 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:34.046883 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:34.048482 (+ 1599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:51:34.048485 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:51:34.048488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:51:34.048513 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:51:34.050216 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:34.050692 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:34.050695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:34.050697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:34.050775 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:51:34.051060 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:51:34.051063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:51:34.051065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:51:34.051251 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:51:34.051339 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:34.051659 (+ 320us) server_negotiation.cc:299] Negotiation successful
0621 07:51:34.051780 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260621 07:51:34.053547 23665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 371e06a939234fd59161dfbd663a8351 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "371e06a939234fd59161dfbd663a8351" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371e06a939234fd59161dfbd663a8351" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34805 } } peers { permanent_uuid: "942def5184454e568c77a7b32b42cffc" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 32909 } } peers { permanent_uuid: "7c3bd96434ac476c875a16578556b4f2" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 44291 } } }
I20260621 07:51:34.053678 23665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 371e06a939234fd59161dfbd663a8351 [sys.catalog]: This master's current role is: LEADER
I20260621 07:51:34.054651 23679 mvcc.cc:205] Tried to move back new op lower bound from 7299187892312821760 to 7299187892271783936. Current Snapshot: MvccSnapshot[applied={T|T < 7299187892312821760}]
I20260621 07:51:34.054891 23625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c3bd96434ac476c875a16578556b4f2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "371e06a939234fd59161dfbd663a8351" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371e06a939234fd59161dfbd663a8351" member_type: VOTER last_known_addr { host: "127.0.192.254" port: 34805 } } peers { permanent_uuid: "942def5184454e568c77a7b32b42cffc" member_type: VOTER last_known_addr { host: "127.0.192.253" port: 32909 } } peers { permanent_uuid: "7c3bd96434ac476c875a16578556b4f2" member_type: VOTER last_known_addr { host: "127.0.192.252" port: 44291 } } }
I20260621 07:51:34.054991 23625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c3bd96434ac476c875a16578556b4f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:51:34.063879 23671 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:51:34.064941 23671 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:51:34.085808 23671 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 371e06a939234fd59161dfbd663a8351: Generated new TSK 0
I20260621 07:51:34.086529 23671 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 07:51:34.113400 23548 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:51:34.110649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46370 (local address 127.0.192.253:32909)
0621 07:51:34.110738 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:34.110742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:34.112394 (+ 1652us) server_negotiation.cc:407] Connection header received
0621 07:51:34.112578 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:34.112595 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:34.112639 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:34.112687 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:34.113292 (+ 605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46370: BlockingRecv error: recv got EOF from 127.0.0.1:46370 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260621 07:51:34.114954 23472 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:51:34.111283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36018 (local address 127.0.192.254:34805)
0621 07:51:34.111370 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:34.111374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:34.113977 (+ 2603us) server_negotiation.cc:407] Connection header received
0621 07:51:34.114338 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:34.114341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:34.114389 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:34.114428 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:34.114842 (+ 414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36018: BlockingRecv error: recv got EOF from 127.0.0.1:36018 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260621 07:51:34.115919 23622 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:51:34.111851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58772 (local address 127.0.192.252:44291)
0621 07:51:34.111955 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:34.111959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:34.114170 (+ 2211us) server_negotiation.cc:407] Connection header received
0621 07:51:34.115328 (+ 1158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:34.115331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:34.115369 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:34.115411 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:34.115828 (+ 417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58772: BlockingRecv error: recv got EOF from 127.0.0.1:58772 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260621 07:51:34.122206 23548 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:51:34.120913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46376 (local address 127.0.192.253:32909)
0621 07:51:34.121004 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:34.121007 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:34.121414 (+ 407us) server_negotiation.cc:407] Connection header received
0621 07:51:34.121598 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:34.121601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:34.121654 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:34.121697 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:34.122124 (+ 427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46376: BlockingRecv error: recv got EOF from 127.0.0.1:46376 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260621 07:51:34.126566 23548 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:51:34.125266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46388 (local address 127.0.192.253:32909)
0621 07:51:34.125346 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:34.125349 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:34.125772 (+ 423us) server_negotiation.cc:407] Connection header received
0621 07:51:34.125917 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:34.125919 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:34.125968 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:34.126015 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:34.126488 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46388: BlockingRecv error: recv got EOF from 127.0.0.1:46388 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260621 07:51:34.131398 23548 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:51:34.130209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46390 (local address 127.0.192.253:32909)
0621 07:51:34.130282 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:34.130286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:34.130686 (+ 400us) server_negotiation.cc:407] Connection header received
0621 07:51:34.130845 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:34.130847 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:34.130882 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:34.130925 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:34.131331 (+ 406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46390: BlockingRecv error: recv got EOF from 127.0.0.1:46390 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260621 07:51:34.138481 23548 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:51:34.137124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46400 (local address 127.0.192.253:32909)
0621 07:51:34.137235 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:34.137239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:34.137725 (+ 486us) server_negotiation.cc:407] Connection header received
0621 07:51:34.137915 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:34.137917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:34.137968 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:34.138012 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:34.138414 (+ 402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46400: BlockingRecv error: recv got EOF from 127.0.0.1:46400 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260621 07:51:34.149524 23548 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:51:34.148178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46402 (local address 127.0.192.253:32909)
0621 07:51:34.148258 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:34.148262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:34.148753 (+ 491us) server_negotiation.cc:407] Connection header received
0621 07:51:34.148954 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:34.148958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:34.148999 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:34.149054 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:34.149457 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46402: BlockingRecv error: recv got EOF from 127.0.0.1:46402 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260621 07:51:34.168946 23548 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:51:34.167530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46410 (local address 127.0.192.253:32909)
0621 07:51:34.167622 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:34.167626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:34.168118 (+ 492us) server_negotiation.cc:407] Connection header received
0621 07:51:34.168277 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:34.168280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:34.168323 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:34.168367 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:34.168852 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46410: BlockingRecv error: recv got EOF from 127.0.0.1:46410 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260621 07:51:34.204198 23548 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:51:34.203128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46422 (local address 127.0.192.253:32909)
0621 07:51:34.203240 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:34.203244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:34.203481 (+ 237us) server_negotiation.cc:407] Connection header received
0621 07:51:34.203547 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:34.203550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:34.203610 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:34.203725 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:34.204093 (+ 368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46422: BlockingRecv error: recv got EOF from 127.0.0.1:46422 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260621 07:51:34.271857 23548 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:51:34.270611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46436 (local address 127.0.192.253:32909)
0621 07:51:34.270748 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:34.270754 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:34.270959 (+ 205us) server_negotiation.cc:407] Connection header received
0621 07:51:34.271123 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:34.271128 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:34.271195 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:34.271284 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:34.271730 (+ 446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46436: BlockingRecv error: recv got EOF from 127.0.0.1:46436 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260621 07:51:34.402974 23548 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:51:34.401864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46446 (local address 127.0.192.253:32909)
0621 07:51:34.401996 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:34.402002 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:34.402160 (+ 158us) server_negotiation.cc:407] Connection header received
0621 07:51:34.402312 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:34.402315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:34.402376 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:34.402456 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:34.402869 (+ 413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46446: BlockingRecv error: recv got EOF from 127.0.0.1:46446 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260621 07:51:34.661949 23548 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:51:34.661072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46458 (local address 127.0.192.253:32909)
0621 07:51:34.661194 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:34.661197 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:34.661277 (+ 80us) server_negotiation.cc:407] Connection header received
0621 07:51:34.661381 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:34.661383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:34.661427 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:34.661500 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:51:34.661862 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46458: BlockingRecv error: recv got EOF from 127.0.0.1:46458 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260621 07:51:34.977165 23636 catalog_manager.cc:1259] Loaded cluster ID: 6d784cd81df042acbed7af9c419c1e60
I20260621 07:51:34.977248 23636 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7c3bd96434ac476c875a16578556b4f2: loading cluster ID for follower catalog manager: success
I20260621 07:51:34.978842 23636 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7c3bd96434ac476c875a16578556b4f2: acquiring CA information for follower catalog manager: success
I20260621 07:51:34.980008 23636 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7c3bd96434ac476c875a16578556b4f2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 07:51:35.001995 23650 catalog_manager.cc:1259] Loaded cluster ID: 6d784cd81df042acbed7af9c419c1e60
I20260621 07:51:35.002058 23650 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 942def5184454e568c77a7b32b42cffc: loading cluster ID for follower catalog manager: success
I20260621 07:51:35.003465 23650 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 942def5184454e568c77a7b32b42cffc: acquiring CA information for follower catalog manager: success
I20260621 07:51:35.004436 23650 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 942def5184454e568c77a7b32b42cffc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 07:51:35.179642 23749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:35.175939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46466 (local address 127.0.192.253:32909)
0621 07:51:35.176226 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:35.176230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:35.176248 (+ 18us) server_negotiation.cc:407] Connection header received
0621 07:51:35.176307 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:35.176309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:35.176466 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:35.176646 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:51:35.176877 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:35.177420 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:35.178231 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:35.178456 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:35.178544 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:35.178815 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:51:35.179308 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:51:35.179412 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:35.179462 (+ 50us) server_negotiation.cc:299] Negotiation successful
0621 07:51:35.179511 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":74,"threads_started":1}
I20260621 07:51:35.185794 23755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:35.182495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58780 (local address 127.0.192.252:44291)
0621 07:51:35.182733 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:35.182737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:35.182751 (+ 14us) server_negotiation.cc:407] Connection header received
0621 07:51:35.182826 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:35.182828 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:35.182936 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:35.183045 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:51:35.183227 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:35.183714 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:35.184476 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:35.184775 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:35.184864 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:35.185102 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:51:35.185481 (+ 379us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:51:35.185576 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:35.185631 (+ 55us) server_negotiation.cc:299] Negotiation successful
0621 07:51:35.185677 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":104,"threads_started":1}
I20260621 07:51:35.192770 23749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:35.188840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46480 (local address 127.0.192.253:32909)
0621 07:51:35.188988 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:35.188991 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:35.189318 (+ 327us) server_negotiation.cc:407] Connection header received
0621 07:51:35.189479 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:35.189483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:35.189531 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:35.189574 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:51:35.189923 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:35.190658 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:35.191810 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:35.192011 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:35.192105 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:35.192332 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:51:35.192402 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:51:35.192457 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:35.192599 (+ 142us) server_negotiation.cc:299] Negotiation successful
0621 07:51:35.192639 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 07:51:35.193475 23762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:35.188959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36020 (local address 127.0.192.254:34805)
0621 07:51:35.189232 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:35.189237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:35.189253 (+ 16us) server_negotiation.cc:407] Connection header received
0621 07:51:35.189433 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:35.189438 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:35.189608 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:35.189754 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:51:35.190028 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:35.190757 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:35.191851 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:35.192156 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:35.192266 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:35.192509 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:51:35.192835 (+ 326us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:51:35.192941 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:35.193237 (+ 296us) server_negotiation.cc:299] Negotiation successful
0621 07:51:35.193294 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":79,"threads_started":1}
I20260621 07:51:35.194643 23755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:51:35.190537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58784 (local address 127.0.192.252:44291)
0621 07:51:35.190651 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0621 07:51:35.190656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:51:35.190866 (+ 210us) server_negotiation.cc:407] Connection header received
0621 07:51:35.191066 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:51:35.191069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:51:35.191140 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:51:35.191207 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:51:35.191569 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:35.192202 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:35.193463 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:51:35.193756 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:51:35.193858 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:51:35.194249 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:51:35.194316 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:51:35.194388 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0621 07:51:35.194481 (+ 93us) server_negotiation.cc:299] Negotiation successful
0621 07:51:35.194522 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 07:51:35.196691 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 23401
I20260621 07:51:35.203339 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 23476
I20260621 07:51:35.208957 771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeKA8lE/build/debug/bin/kudu with pid 23551
2026-06-21T07:51:35Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2898 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5809 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (266682 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST